[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:36:50 dist-test-slave-drl0 krb5kdc[4046](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:36:50 dist-test-slave-drl0 krb5kdc[4046](info): set up 2 sockets
Aug 10 11:36:50 dist-test-slave-drl0 krb5kdc[4046](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260810 11:36:51.489063 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.458s user 0.007s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:36:51 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:36:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:36:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:36:51.703105 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35673
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35673
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:52.411151 4068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:52.411919 4068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:52.412204 4068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:52.425012 4068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:36:52.425239 4068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:52.425354 4068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:52.425487 4068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:36:52.425623 4068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:36:52.441560 4068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35673
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:52.444749 4068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:52.447384 4068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:52.475826 4079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:52.475675 4082 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:52.476532 4078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:52.477713 4068 server_base.cc:1034] running on GCE node
I20260810 11:36:52.479532 4068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:52.482494 4068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:52.485973 4068 hybrid_clock.cc:648] HybridClock initialized: now 1786361812485777 us; error 106 us; skew 500 ppm
I20260810 11:36:52.486765 4068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:52 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361812, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:52.492020 4068 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:36:52.495054 4068 webserver.cc:460] Webserver started at http://127.3.240.254:33057/ using document root <none> and password file <none>
I20260810 11:36:52.496290 4068 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:52.496541 4068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:52.497452 4068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:52.504161 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "a39b363824bf48ac819344746a650853"
format_stamp: "Formatted at 2026-08-10 11:36:52 on dist-test-slave-drl0"
server_key: "160e457c4fd44f0b75d84f78838baea5"
server_key_iv: "e3bfe7748f793c6b82c79d92d5fe7fdc"
server_key_version: "encryptionkey@0"
I20260810 11:36:52.505684 4068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a39b363824bf48ac819344746a650853"
format_stamp: "Formatted at 2026-08-10 11:36:52 on dist-test-slave-drl0"
server_key: "160e457c4fd44f0b75d84f78838baea5"
server_key_iv: "e3bfe7748f793c6b82c79d92d5fe7fdc"
server_key_version: "encryptionkey@0"
I20260810 11:36:52.515817 4068 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.006s sys 0.005s
I20260810 11:36:52.523204 4093 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:52.525595 4068 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.003s sys 0.000s
I20260810 11:36:52.525882 4068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "a39b363824bf48ac819344746a650853"
format_stamp: "Formatted at 2026-08-10 11:36:52 on dist-test-slave-drl0"
server_key: "160e457c4fd44f0b75d84f78838baea5"
server_key_iv: "e3bfe7748f793c6b82c79d92d5fe7fdc"
server_key_version: "encryptionkey@0"
I20260810 11:36:52.526232 4068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:52.551975 4068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:52.559839 4068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:52.560539 4068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:52.616230 4068 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35673
I20260810 11:36:52.616285 4165 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35673 every 8 connection(s)
I20260810 11:36:52.619518 4068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:36:52.625216 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4068
I20260810 11:36:52.625777 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:36:52.628419 4035 external_mini_cluster.cc:1428] Setting key 3c246f5665fe65215ff26552a9a1848f
I20260810 11:36:52.628432 4166 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:36:52.651356 4166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853: Bootstrap starting.
I20260810 11:36:52.658138 4166 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853: Neither blocks nor log segments found. Creating new log.
I20260810 11:36:52.660604 4166 log.cc:824] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853: Log is configured to *not* fsync() on all Append() calls
I20260810 11:36:52.666360 4166 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853: No bootstrap required, opened a new log
Aug 10 11:36:52 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361811, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:52.681622 4166 raft_consensus.cc:348] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } }
I20260810 11:36:52.682340 4166 raft_consensus.cc:374] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:36:52.682578 4166 raft_consensus.cc:729] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39b363824bf48ac819344746a650853, State: Initialized, Role: FOLLOWER
I20260810 11:36:52.684348 4166 consensus_queue.cc:260] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } }
I20260810 11:36:52.684991 4166 raft_consensus.cc:388] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:36:52.685542 4166 raft_consensus.cc:482] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:36:52.686048 4166 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:36:52.690083 4166 raft_consensus.cc:504] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } }
I20260810 11:36:52.690862 4169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:52.636242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53384 (local address 127.3.240.254:35673)
0810 11:36:52.638523 (+ 2281us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:52.638539 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:52.641068 (+ 2529us) server_negotiation.cc:407] Connection header received
0810 11:36:52.656113 (+ 15045us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:52.656137 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:52.656731 (+ 594us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:52.657309 (+ 578us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:52.662201 (+ 4892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:52.665142 (+ 2941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:52.667828 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:52.668823 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:52.669165 (+ 342us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:52.677248 (+ 8083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:52.677335 (+ 87us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:52.677374 (+ 39us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:52.677417 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:52.682730 (+ 5313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:52.684004 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:52.684011 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:52.684017 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:52.684500 (+ 483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:52.685160 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:52.685167 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:52.685170 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:52.686078 (+ 908us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:52.686699 (+ 621us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:52.687751 (+ 1052us) server_negotiation.cc:299] Negotiation successful
0810 11:36:52.688325 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1805,"thread_start_us":1255,"threads_started":1}
I20260810 11:36:52.691562 4166 leader_election.cc:302] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a39b363824bf48ac819344746a650853; no voters:
I20260810 11:36:52.693259 4166 leader_election.cc:288] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:36:52.693555 4174 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:36:52.697997 4166 sys_catalog.cc:564] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:36:52.699620 4174 raft_consensus.cc:686] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [term 1 LEADER]: Becoming Leader. State: Replica: a39b363824bf48ac819344746a650853, State: Running, Role: LEADER
I20260810 11:36:52.701054 4174 consensus_queue.cc:237] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } }
I20260810 11:36:52.711987 4176 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a39b363824bf48ac819344746a650853. Latest consensus state: current_term: 1 leader_uuid: "a39b363824bf48ac819344746a650853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } } }
I20260810 11:36:52.712709 4176 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [sys.catalog]: This master's current role is: LEADER
I20260810 11:36:52.713657 4175 sys_catalog.cc:455] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a39b363824bf48ac819344746a650853" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a39b363824bf48ac819344746a650853" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35673 } } }
I20260810 11:36:52.714339 4175 sys_catalog.cc:458] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853 [sys.catalog]: This master's current role is: LEADER
I20260810 11:36:52.722362 4190 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:36:52.733848 4190 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:36:52.754383 4190 catalog_manager.cc:1347] Generated new cluster ID: 353262b280b04c7bbd17cbb5e6e3cd65
I20260810 11:36:52.754742 4190 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:36:52.786780 4190 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:36:52.789548 4190 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:36:52.807866 4190 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a39b363824bf48ac819344746a650853: Generated new TSK 0
I20260810 11:36:52.809028 4190 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:36:52.900197 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:53.641191 4205 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:53.641620 4205 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:53.641896 4205 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:53.652602 4205 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:53.652849 4205 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:53.653122 4205 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:36:53.667631 4205 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:53.669459 4205 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:53.671052 4205 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:53.692762 4212 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:53.693002 4213 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:53.694339 4217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:53.696117 4205 server_base.cc:1034] running on GCE node
I20260810 11:36:53.696914 4205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:53.698427 4205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:53.699721 4205 hybrid_clock.cc:648] HybridClock initialized: now 1786361813699643 us; error 54 us; skew 500 ppm
I20260810 11:36:53.700227 4205 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:53 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:53.705667 4205 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:36:53.708019 4205 webserver.cc:460] Webserver started at http://127.3.240.193:39473/ using document root <none> and password file <none>
I20260810 11:36:53.708881 4205 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:53.709030 4205 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:53.709451 4205 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:53.712862 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "58a7e307427c46748741e94f7f350378"
format_stamp: "Formatted at 2026-08-10 11:36:53 on dist-test-slave-drl0"
server_key: "db6d087e4c3c795c3d1541606a6ed993"
server_key_iv: "4d4c983cb6882e1b895150b7476f626b"
server_key_version: "encryptionkey@0"
I20260810 11:36:53.714529 4205 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "58a7e307427c46748741e94f7f350378"
format_stamp: "Formatted at 2026-08-10 11:36:53 on dist-test-slave-drl0"
server_key: "db6d087e4c3c795c3d1541606a6ed993"
server_key_iv: "4d4c983cb6882e1b895150b7476f626b"
server_key_version: "encryptionkey@0"
I20260810 11:36:53.723989 4205 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260810 11:36:53.729382 4230 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:53.730904 4205 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260810 11:36:53.731180 4205 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "58a7e307427c46748741e94f7f350378"
format_stamp: "Formatted at 2026-08-10 11:36:53 on dist-test-slave-drl0"
server_key: "db6d087e4c3c795c3d1541606a6ed993"
server_key_iv: "4d4c983cb6882e1b895150b7476f626b"
server_key_version: "encryptionkey@0"
I20260810 11:36:53.731560 4205 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:53.752365 4205 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:53.759665 4205 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:53.760041 4205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:53.761482 4205 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:36:53.763995 4205 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:36:53.764174 4205 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:53.764405 4205 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:36:53.764493 4205 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:53.857510 4205 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:42275
I20260810 11:36:53.857622 4399 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:42275 every 8 connection(s)
I20260810 11:36:53.859292 4205 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:36:53.861640 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4205
I20260810 11:36:53.861953 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:36:53.863142 4035 external_mini_cluster.cc:1428] Setting key f147225466165376173f6b4a4044f3b9
Aug 10 11:36:53 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361813, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:53.891633 4403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:53.864416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:36111 (local address 127.3.240.254:35673)
0810 11:36:53.864997 (+ 581us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:53.865004 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:53.866958 (+ 1954us) server_negotiation.cc:407] Connection header received
0810 11:36:53.869040 (+ 2082us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:53.869045 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:53.869138 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:53.869262 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:53.871400 (+ 2138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:53.872467 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:53.874635 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:53.875715 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:53.875975 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:53.882073 (+ 6098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:53.882082 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:53.882086 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:53.882132 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:53.887195 (+ 5063us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:53.888196 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:53.888201 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:53.888204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:53.888442 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:53.889064 (+ 622us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:53.889068 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:53.889070 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:53.889410 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:53.889662 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:53.890490 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 11:36:53.890843 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":996,"threads_started":1}
I20260810 11:36:53.892681 4404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:53.865218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35673 (local address 127.3.240.193:36111)
0810 11:36:53.866730 (+ 1512us) negotiation.cc:107] Waiting for socket to connect
0810 11:36:53.866770 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:36:53.868760 (+ 1990us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:36:53.869596 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:36:53.869610 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:36:53.870298 (+ 688us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:36:53.871220 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:53.871241 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:53.872671 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:53.872675 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:53.874488 (+ 1813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:53.874501 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:53.876306 (+ 1805us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:53.876319 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:53.876782 (+ 463us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:53.877852 (+ 1070us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:36:53.877883 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:36:53.881572 (+ 3689us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:36:53.887410 (+ 5838us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:53.887421 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:53.887445 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:53.888039 (+ 594us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:53.888574 (+ 535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:53.888579 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:53.888583 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:53.888898 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:53.889571 (+ 673us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:36:53.889580 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:36:53.890056 (+ 476us) client_negotiation.cc:769] Sending connection context
0810 11:36:53.890393 (+ 337us) client_negotiation.cc:240] Negotiation successful
0810 11:36:53.891191 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1213,"thread_start_us":741,"threads_started":1}
I20260810 11:36:53.894836 4400 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35673
I20260810 11:36:53.895623 4400 heartbeater.cc:464] Registering TS with master...
I20260810 11:36:53.896697 4400 heartbeater.cc:511] Master 127.3.240.254:35673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260810 11:36:53.901863 4122 ts_manager.cc:194] Registered new tserver with Master: 58a7e307427c46748741e94f7f350378 (127.3.240.193:42275)
I20260810 11:36:53.904366 4122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:36111
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:36:53.935235 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:54.559010 4408 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:54.559617 4408 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:54.559909 4408 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:54.571075 4408 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:54.571316 4408 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:54.571591 4408 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:36:54.584638 4408 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:54.586223 4408 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:54.587787 4408 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:54.606257 4419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:54.606572 4420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:54.606789 4422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:54.608708 4408 server_base.cc:1034] running on GCE node
I20260810 11:36:54.609790 4408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:54.611549 4408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:54.612947 4408 hybrid_clock.cc:648] HybridClock initialized: now 1786361814612803 us; error 50 us; skew 500 ppm
I20260810 11:36:54.613484 4408 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:54 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:54.617908 4408 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:36:54.621003 4408 webserver.cc:460] Webserver started at http://127.3.240.194:43219/ using document root <none> and password file <none>
I20260810 11:36:54.621922 4408 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:54.622149 4408 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:54.622602 4408 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:54.626754 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "42c78df5a06544d19f770e4295662495"
format_stamp: "Formatted at 2026-08-10 11:36:54 on dist-test-slave-drl0"
server_key: "a8e14f8daade783dcc793e2e5da1f4c5"
server_key_iv: "849ce1c15f5fb5875a0fce72ad86bf7e"
server_key_version: "encryptionkey@0"
I20260810 11:36:54.628332 4408 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "42c78df5a06544d19f770e4295662495"
format_stamp: "Formatted at 2026-08-10 11:36:54 on dist-test-slave-drl0"
server_key: "a8e14f8daade783dcc793e2e5da1f4c5"
server_key_iv: "849ce1c15f5fb5875a0fce72ad86bf7e"
server_key_version: "encryptionkey@0"
I20260810 11:36:54.637346 4408 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.000s sys 0.009s
I20260810 11:36:54.642843 4431 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:54.644909 4408 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260810 11:36:54.645220 4408 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "42c78df5a06544d19f770e4295662495"
format_stamp: "Formatted at 2026-08-10 11:36:54 on dist-test-slave-drl0"
server_key: "a8e14f8daade783dcc793e2e5da1f4c5"
server_key_iv: "849ce1c15f5fb5875a0fce72ad86bf7e"
server_key_version: "encryptionkey@0"
I20260810 11:36:54.645501 4408 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:54.660286 4408 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:54.672254 4408 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:54.672909 4408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:54.674635 4408 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:36:54.676983 4408 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:36:54.677095 4408 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:54.677304 4408 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:36:54.677431 4408 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:54.786091 4408 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39747
I20260810 11:36:54.786267 4596 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39747 every 8 connection(s)
I20260810 11:36:54.788317 4408 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:36:54.799127 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4408
I20260810 11:36:54.799793 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:36:54.800704 4035 external_mini_cluster.cc:1428] Setting key 82cb65a780f45217e6531404778bdeef
Aug 10 11:36:54 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361814, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:54.829428 4600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:54.792861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:37543 (local address 127.3.240.254:35673)
0810 11:36:54.793780 (+ 919us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:54.793786 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:54.795498 (+ 1712us) server_negotiation.cc:407] Connection header received
0810 11:36:54.798683 (+ 3185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:54.798688 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:54.798852 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:54.799045 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:54.802760 (+ 3715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:54.804540 (+ 1780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:54.808042 (+ 3502us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:54.809227 (+ 1185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:54.809747 (+ 520us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:54.817616 (+ 7869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:54.817628 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:54.817634 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:54.817753 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:54.823798 (+ 6045us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:54.825434 (+ 1636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:54.825445 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:54.825452 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:54.825709 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:54.826465 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:54.826474 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:54.826480 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:54.827080 (+ 600us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:54.827509 (+ 429us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:54.828530 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0810 11:36:54.828868 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":781,"thread_start_us":547,"threads_started":1}
I20260810 11:36:54.830436 4604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:54.793489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35673 (local address 127.3.240.194:37543)
0810 11:36:54.795315 (+ 1826us) negotiation.cc:107] Waiting for socket to connect
0810 11:36:54.795363 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:36:54.798118 (+ 2755us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:36:54.799713 (+ 1595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:36:54.799731 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:36:54.801220 (+ 1489us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:36:54.802542 (+ 1322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:54.802567 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:54.804828 (+ 2261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:54.804833 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:54.807844 (+ 3011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:54.807864 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:54.809774 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:54.809784 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:54.810234 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:54.811951 (+ 1717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:36:54.811986 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:36:54.817183 (+ 5197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:36:54.823993 (+ 6810us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:54.824001 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:54.824017 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:54.825153 (+ 1136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:54.825946 (+ 793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:54.825949 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:54.825952 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:54.826293 (+ 341us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:54.827564 (+ 1271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:36:54.827572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:36:54.828032 (+ 460us) client_negotiation.cc:769] Sending connection context
0810 11:36:54.828326 (+ 294us) client_negotiation.cc:240] Negotiation successful
0810 11:36:54.828954 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":876,"threads_started":1}
I20260810 11:36:54.833083 4598 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35673
I20260810 11:36:54.833478 4598 heartbeater.cc:464] Registering TS with master...
I20260810 11:36:54.834587 4598 heartbeater.cc:511] Master 127.3.240.254:35673 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260810 11:36:54.836573 4122 ts_manager.cc:194] Registered new tserver with Master: 42c78df5a06544d19f770e4295662495 (127.3.240.194:39747)
I20260810 11:36:54.837760 4122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:37543
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:36:54.879707 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45939
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:36:54.909250 4400 heartbeater.cc:503] Master 127.3.240.254:35673 was elected leader, sending a full tablet report...
W20260810 11:36:55.538506 4608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:55.539079 4608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:55.539230 4608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:55.550616 4608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:55.550773 4608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:55.551007 4608 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:36:55.565186 4608 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45939
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35673
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:55.566859 4608 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:55.568634 4608 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:55.590348 4614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:55.590220 4615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:55.590469 4618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:55.591389 4608 server_base.cc:1034] running on GCE node
I20260810 11:36:55.592502 4608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:55.593914 4608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:55.595273 4608 hybrid_clock.cc:648] HybridClock initialized: now 1786361815595138 us; error 50 us; skew 500 ppm
I20260810 11:36:55.595779 4608 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:55.600925 4608 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:36:55.604101 4608 webserver.cc:460] Webserver started at http://127.3.240.195:34013/ using document root <none> and password file <none>
I20260810 11:36:55.605464 4608 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:55.605880 4608 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:55.606571 4608 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:55.611917 4608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "6b1d83e2cb9f4759b018e2f3e465a673"
format_stamp: "Formatted at 2026-08-10 11:36:55 on dist-test-slave-drl0"
server_key: "ff043b02ef0f2fa74bbc9c4ad4daad8f"
server_key_iv: "2a968c15df00086ba12057dc21e4fb31"
server_key_version: "encryptionkey@0"
I20260810 11:36:55.613166 4608 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "6b1d83e2cb9f4759b018e2f3e465a673"
format_stamp: "Formatted at 2026-08-10 11:36:55 on dist-test-slave-drl0"
server_key: "ff043b02ef0f2fa74bbc9c4ad4daad8f"
server_key_iv: "2a968c15df00086ba12057dc21e4fb31"
server_key_version: "encryptionkey@0"
I20260810 11:36:55.621946 4608 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260810 11:36:55.628507 4634 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:55.630749 4608 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260810 11:36:55.631487 4608 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "6b1d83e2cb9f4759b018e2f3e465a673"
format_stamp: "Formatted at 2026-08-10 11:36:55 on dist-test-slave-drl0"
server_key: "ff043b02ef0f2fa74bbc9c4ad4daad8f"
server_key_iv: "2a968c15df00086ba12057dc21e4fb31"
server_key_version: "encryptionkey@0"
I20260810 11:36:55.631798 4608 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:55.656858 4608 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:55.663851 4608 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:55.664304 4608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:55.666112 4608 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:36:55.669349 4608 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:36:55.669721 4608 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.001s user 0.000s sys 0.000s
I20260810 11:36:55.670253 4608 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:36:55.670573 4608 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:55.790297 4608 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43067
I20260810 11:36:55.790370 4781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43067 every 8 connection(s)
I20260810 11:36:55.793213 4608 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:36:55.802554 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4608
I20260810 11:36:55.803083 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:36:55.804062 4035 external_mini_cluster.cc:1428] Setting key d52e1128c525058d6196b660fef087a5
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:55.837873 4786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:55.799018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47513 (local address 127.3.240.254:35673)
0810 11:36:55.799646 (+ 628us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:55.799652 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:55.807396 (+ 7744us) server_negotiation.cc:407] Connection header received
0810 11:36:55.807835 (+ 439us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:55.807840 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:55.807938 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:55.808053 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:55.812179 (+ 4126us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.814424 (+ 2245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.818187 (+ 3763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.819323 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.819596 (+ 273us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:55.826757 (+ 7161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:55.826766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:55.826771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:55.826823 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:55.831627 (+ 4804us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:55.833481 (+ 1854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:55.833489 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:55.833495 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:55.833702 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:55.834850 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:55.834884 (+ 34us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:55.834891 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:55.835425 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:55.835624 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:55.836913 (+ 1289us) server_negotiation.cc:299] Negotiation successful
0810 11:36:55.837181 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":507,"thread_start_us":365,"threads_started":1}
I20260810 11:36:55.839376 4787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:55.800585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35673 (local address 127.3.240.195:47513)
0810 11:36:55.803553 (+ 2968us) negotiation.cc:107] Waiting for socket to connect
0810 11:36:55.803634 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0810 11:36:55.807449 (+ 3815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:36:55.808553 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:36:55.808568 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:36:55.810099 (+ 1531us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:36:55.811727 (+ 1628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:55.811777 (+ 50us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.814819 (+ 3042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.814827 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:55.817900 (+ 3073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:55.817926 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.820754 (+ 2828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.820764 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:55.821118 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:55.821926 (+ 808us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:36:55.821952 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:36:55.825657 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:36:55.831789 (+ 6132us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:55.831801 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:55.831821 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:55.833292 (+ 1471us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:55.834030 (+ 738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:55.834037 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:55.834044 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:55.834676 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:55.835727 (+ 1051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:36:55.835740 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:36:55.836343 (+ 603us) client_negotiation.cc:769] Sending connection context
0810 11:36:55.836780 (+ 437us) client_negotiation.cc:240] Negotiation successful
0810 11:36:55.837514 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2349,"spinlock_wait_cycles":29440,"thread_start_us":2104,"threads_started":1}
I20260810 11:36:55.841486 4782 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35673
I20260810 11:36:55.841899 4782 heartbeater.cc:464] Registering TS with master...
I20260810 11:36:55.842437 4598 heartbeater.cc:503] Master 127.3.240.254:35673 was elected leader, sending a full tablet report...
I20260810 11:36:55.843062 4782 heartbeater.cc:511] Master 127.3.240.254:35673 requested a full tablet report, sending...
I20260810 11:36:55.845361 4122 ts_manager.cc:194] Registered new tserver with Master: 6b1d83e2cb9f4759b018e2f3e465a673 (127.3.240.195:43067)
I20260810 11:36:55.846235 4122 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:47513
I20260810 11:36:55.850646 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:36:55.939314 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:55.915366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44598 (local address 127.3.240.193:42275)
0810 11:36:55.916243 (+ 877us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:55.916251 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:55.916511 (+ 260us) server_negotiation.cc:407] Connection header received
0810 11:36:55.916808 (+ 297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:55.916815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:55.917056 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:55.917213 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:55.920830 (+ 3617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.921980 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.923898 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.924544 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.924909 (+ 365us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:55.931983 (+ 7074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:55.932003 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:55.932019 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:55.932067 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:55.935571 (+ 3504us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:55.936344 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:55.936352 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:55.936355 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:55.936566 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:55.937066 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:55.937071 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:55.937076 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:55.937668 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:55.938210 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:55.938407 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 11:36:55.938818 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":483,"threads_started":1}
W20260810 11:36:55.940763 4281 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44598
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 11:36:55 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:36:56.009217 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:55.987332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44600 (local address 127.3.240.193:42275)
0810 11:36:55.987820 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:55.987831 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:55.988354 (+ 523us) server_negotiation.cc:407] Connection header received
0810 11:36:55.988536 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:55.988540 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:55.988614 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:55.988712 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:55.991370 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.992537 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.994563 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:55.995160 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:55.995285 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:56.001414 (+ 6129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:56.001421 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:56.001425 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:56.001485 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:56.004826 (+ 3341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.006308 (+ 1482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.006314 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.006317 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.006482 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.007387 (+ 905us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.007392 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.007394 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.007796 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:56.008020 (+ 224us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:56.008672 (+ 652us) server_negotiation.cc:299] Negotiation successful
0810 11:36:56.008935 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120}
I20260810 11:36:56.012560 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4205
I20260810 11:36:56.034731 4792 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:56.015042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44602 (local address 127.3.240.193:42275)
0810 11:36:56.015275 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:56.015281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:56.015306 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:36:56.015350 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:56.015354 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:56.015437 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:56.015546 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:56.017169 (+ 1623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.017993 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.019491 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.020180 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.020384 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:56.024350 (+ 3966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:56.024358 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:56.024362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:56.024418 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:56.029969 (+ 5551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.031219 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.031226 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.031230 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.031528 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.032370 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.032376 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.032379 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.032827 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:56.033412 (+ 585us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:56.033769 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 11:36:56.034037 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:36:56.035887 4392 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:36:56.132319 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4205
I20260810 11:36:56.158351 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4408
Aug 10 11:36:56 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:36:56.180369 4809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:56.159526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34676 (local address 127.3.240.194:39747)
0810 11:36:56.160537 (+ 1011us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:56.160545 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:56.160572 (+ 27us) server_negotiation.cc:407] Connection header received
0810 11:36:56.160635 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:56.160642 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:56.160847 (+ 205us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:56.161002 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:56.163044 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.164791 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.166555 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.167276 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.167399 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:56.173510 (+ 6111us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:56.173529 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:56.173551 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:56.173613 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:56.177229 (+ 3616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.178054 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.178061 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.178064 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.178199 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.178607 (+ 408us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.178612 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.178616 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.179046 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:56.179359 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:56.179706 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 11:36:56.179991 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":833,"thread_start_us":639,"threads_started":1}
I20260810 11:36:56.182255 4591 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:36:56.273914 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4408
I20260810 11:36:56.298085 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4608
Aug 10 11:36:56 dist-test-slave-drl0 krb5kdc[4046](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361815, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:36:56.318280 4812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:56.299523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43868 (local address 127.3.240.195:43067)
0810 11:36:56.300143 (+ 620us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:56.300151 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:56.300184 (+ 33us) server_negotiation.cc:407] Connection header received
0810 11:36:56.300255 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:56.300261 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:56.300484 (+ 223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:56.300626 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:56.302090 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.303560 (+ 1470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.305287 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:56.305855 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:56.305970 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:56.310840 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:56.310864 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:56.310886 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:56.311166 (+ 280us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:56.315177 (+ 4011us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.316013 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.316032 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.316035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.316220 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:56.316676 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:56.316681 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:56.316683 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:56.317102 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:56.317357 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:56.317717 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 11:36:56.317969 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":304,"threads_started":1}
I20260810 11:36:56.319703 4776 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:36:56.425834 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4608
I20260810 11:36:56.452492 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4068
I20260810 11:36:56.454253 4159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:36:56.571314 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4068
2026-08-10T11:36:56Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (6623 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:36:56 dist-test-slave-drl0 krb5kdc[4823](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:36:56 dist-test-slave-drl0 krb5kdc[4823](info): set up 2 sockets
Aug 10 11:36:56 dist-test-slave-drl0 krb5kdc[4823](info): commencing operation
krb5kdc: starting...
W20260810 11:36:57.942761 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.308s user 0.008s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:36:58 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:36:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:36:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:36:58.195655 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43103
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43103
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:58.567596 4850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:58.567909 4850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:58.568055 4850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:58.575857 4850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:36:58.576026 4850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:58.576149 4850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:58.576257 4850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:36:58.576327 4850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:36:58.588608 4850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43103
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43103
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:58.590368 4850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:58.591831 4850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:58.604851 4860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:58.604799 4861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:58.604969 4863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:58.606571 4850 server_base.cc:1034] running on GCE node
I20260810 11:36:58.608112 4850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:58.609777 4850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:58.611089 4850 hybrid_clock.cc:648] HybridClock initialized: now 1786361818611040 us; error 66 us; skew 500 ppm
I20260810 11:36:58.611464 4850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:58 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361818, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:58.618486 4850 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:36:58.620282 4850 webserver.cc:460] Webserver started at http://127.3.240.254:32999/ using document root <none> and password file <none>
I20260810 11:36:58.621083 4850 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:58.621191 4850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:58.621471 4850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:58.624531 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "deebbd75ff3a4a54a0bdcc9182b6344a"
format_stamp: "Formatted at 2026-08-10 11:36:58 on dist-test-slave-drl0"
server_key: "84dbc7cf3e53cd0cec518fa43cf3656d"
server_key_iv: "c459cb9ec555f6630bbda28817e256cb"
server_key_version: "encryptionkey@0"
I20260810 11:36:58.625216 4850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "deebbd75ff3a4a54a0bdcc9182b6344a"
format_stamp: "Formatted at 2026-08-10 11:36:58 on dist-test-slave-drl0"
server_key: "84dbc7cf3e53cd0cec518fa43cf3656d"
server_key_iv: "c459cb9ec555f6630bbda28817e256cb"
server_key_version: "encryptionkey@0"
I20260810 11:36:58.631249 4850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260810 11:36:58.635281 4875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:58.636917 4850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:36:58.637121 4850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "deebbd75ff3a4a54a0bdcc9182b6344a"
format_stamp: "Formatted at 2026-08-10 11:36:58 on dist-test-slave-drl0"
server_key: "84dbc7cf3e53cd0cec518fa43cf3656d"
server_key_iv: "c459cb9ec555f6630bbda28817e256cb"
server_key_version: "encryptionkey@0"
I20260810 11:36:58.637338 4850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:58.649304 4850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:58.654309 4850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:58.654819 4850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:58.678992 4850 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43103
I20260810 11:36:58.679160 4947 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43103 every 8 connection(s)
I20260810 11:36:58.680716 4850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:36:58.685951 4948 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:36:58.690828 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4850
I20260810 11:36:58.691154 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:36:58.692116 4035 external_mini_cluster.cc:1428] Setting key aef1ede51479e726c67ba58e16d94f47
I20260810 11:36:58.702350 4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a: Bootstrap starting.
I20260810 11:36:58.706342 4948 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a: Neither blocks nor log segments found. Creating new log.
I20260810 11:36:58.707923 4948 log.cc:824] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:36:58 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:58.712438 4948 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a: No bootstrap required, opened a new log
I20260810 11:36:58.720602 4953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:58.694889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58520 (local address 127.3.240.254:43103)
0810 11:36:58.695786 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:58.695798 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:58.695919 (+ 121us) server_negotiation.cc:407] Connection header received
0810 11:36:58.697726 (+ 1807us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:58.697749 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:58.698165 (+ 416us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:58.698511 (+ 346us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:58.700612 (+ 2101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:58.701990 (+ 1378us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:58.703809 (+ 1819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:58.704513 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:58.704650 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:58.711447 (+ 6797us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:58.711458 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:58.711473 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:58.711509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:58.715333 (+ 3824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:58.716229 (+ 896us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:58.716236 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:58.716242 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:58.716520 (+ 278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:58.716971 (+ 451us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:58.716975 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:58.716977 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:58.717571 (+ 594us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:58.718113 (+ 542us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:58.718571 (+ 458us) server_negotiation.cc:299] Negotiation successful
0810 11:36:58.719342 (+ 771us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":398,"threads_started":1}
I20260810 11:36:58.721833 4948 raft_consensus.cc:348] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } }
I20260810 11:36:58.722649 4948 raft_consensus.cc:374] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:36:58.723238 4948 raft_consensus.cc:729] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: deebbd75ff3a4a54a0bdcc9182b6344a, State: Initialized, Role: FOLLOWER
I20260810 11:36:58.724797 4948 consensus_queue.cc:260] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } }
I20260810 11:36:58.725216 4948 raft_consensus.cc:388] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:36:58.725389 4948 raft_consensus.cc:482] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:36:58.725572 4948 raft_consensus.cc:3037] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:36:58.730731 4948 raft_consensus.cc:504] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } }
I20260810 11:36:58.732726 4948 leader_election.cc:302] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: deebbd75ff3a4a54a0bdcc9182b6344a; no voters:
I20260810 11:36:58.733404 4948 leader_election.cc:288] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:36:58.733566 4956 raft_consensus.cc:2781] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:36:58.735710 4956 raft_consensus.cc:686] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [term 1 LEADER]: Becoming Leader. State: Replica: deebbd75ff3a4a54a0bdcc9182b6344a, State: Running, Role: LEADER
I20260810 11:36:58.736421 4948 sys_catalog.cc:564] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:36:58.736496 4956 consensus_queue.cc:237] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } }
I20260810 11:36:58.739907 4958 sys_catalog.cc:455] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [sys.catalog]: SysCatalogTable state changed. Reason: New leader deebbd75ff3a4a54a0bdcc9182b6344a. Latest consensus state: current_term: 1 leader_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } } }
I20260810 11:36:58.741236 4958 sys_catalog.cc:458] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [sys.catalog]: This master's current role is: LEADER
I20260810 11:36:58.740224 4957 sys_catalog.cc:455] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "deebbd75ff3a4a54a0bdcc9182b6344a" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43103 } } }
I20260810 11:36:58.742056 4957 sys_catalog.cc:458] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a [sys.catalog]: This master's current role is: LEADER
I20260810 11:36:58.744784 4974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:36:58.749547 4974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:36:58.759550 4974 catalog_manager.cc:1347] Generated new cluster ID: 47a4dbd2052e43b4b5b95b5b2adf7faa
I20260810 11:36:58.759690 4974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:36:58.789376 4974 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:36:58.791663 4974 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:36:58.802909 4974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P deebbd75ff3a4a54a0bdcc9182b6344a: Generated new TSK 0
I20260810 11:36:58.804757 4974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:36:58.882877 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:59.259627 4985 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:59.259930 4985 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:59.260105 4985 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:59.269229 4985 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:59.269361 4985 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:59.269562 4985 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:36:59.281025 4985 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:59.282467 4985 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:59.283902 4985 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:59.298575 4995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:59.298669 4997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:59.298621 4993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:59.299614 4985 server_base.cc:1034] running on GCE node
I20260810 11:36:59.300547 4985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:59.301582 4985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:59.302827 4985 hybrid_clock.cc:648] HybridClock initialized: now 1786361819302777 us; error 46 us; skew 500 ppm
I20260810 11:36:59.303268 4985 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:59 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:59.306734 4985 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:36:59.308512 4985 webserver.cc:460] Webserver started at http://127.3.240.193:39027/ using document root <none> and password file <none>
I20260810 11:36:59.309233 4985 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:59.309491 4985 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:59.309872 4985 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:59.312983 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "3be4cef8712e4687a6227b5633f0c88a"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "21e86f4bd4cf94e444f6c0b2312a6c10"
server_key_iv: "91ecf2cfb055dff464a69cb1e53b0b2c"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.313808 4985 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "3be4cef8712e4687a6227b5633f0c88a"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "21e86f4bd4cf94e444f6c0b2312a6c10"
server_key_iv: "91ecf2cfb055dff464a69cb1e53b0b2c"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.320426 4985 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 11:36:59.324586 5005 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.326157 4985 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:36:59.326354 4985 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "3be4cef8712e4687a6227b5633f0c88a"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "21e86f4bd4cf94e444f6c0b2312a6c10"
server_key_iv: "91ecf2cfb055dff464a69cb1e53b0b2c"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.326560 4985 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:59.349251 4985 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:59.354167 4985 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:59.354511 4985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:59.355751 4985 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:36:59.357364 4985 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:36:59.357442 4985 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.357584 4985 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:36:59.357649 4985 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.400516 4985 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:35949
I20260810 11:36:59.400687 5163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:35949 every 8 connection(s)
I20260810 11:36:59.401964 4985 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:36:59.410260 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4985
I20260810 11:36:59.410524 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:36:59.411260 4035 external_mini_cluster.cc:1428] Setting key 0bc24561fee5bece6edcea981b00463a
Aug 10 11:36:59 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:59.428699 5168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:59.404936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55257 (local address 127.3.240.254:43103)
0810 11:36:59.405438 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:59.405443 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:59.406671 (+ 1228us) server_negotiation.cc:407] Connection header received
0810 11:36:59.409258 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:59.409265 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:59.409368 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:59.409536 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:59.411323 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.412187 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.414226 (+ 2039us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.414902 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.415092 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:59.420101 (+ 5009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:59.420110 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:59.420114 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:59.420166 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:59.424369 (+ 4203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:59.425223 (+ 854us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:59.425232 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:59.425239 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:59.425403 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:59.425969 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:59.425977 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:59.425983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:59.426405 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:59.426589 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:59.427924 (+ 1335us) server_negotiation.cc:299] Negotiation successful
0810 11:36:59.428202 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":268,"threads_started":1}
I20260810 11:36:59.430220 5172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:59.405434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43103 (local address 127.3.240.193:55257)
0810 11:36:59.406416 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 11:36:59.406478 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 11:36:59.408904 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:36:59.409791 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:36:59.409799 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:36:59.410352 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:36:59.411136 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:59.411158 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.412424 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.412430 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:59.414041 (+ 1611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:59.414054 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.415347 (+ 1293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.415360 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:59.415593 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:59.416490 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:36:59.416533 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:36:59.419712 (+ 3179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:36:59.424566 (+ 4854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:59.424574 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:59.424588 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:59.425074 (+ 486us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:59.425586 (+ 512us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:59.425590 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:59.425593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:59.425801 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:59.426677 (+ 876us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:36:59.426688 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:36:59.427366 (+ 678us) client_negotiation.cc:769] Sending connection context
0810 11:36:59.427734 (+ 368us) client_negotiation.cc:240] Negotiation successful
0810 11:36:59.428593 (+ 859us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":349,"threads_started":1}
I20260810 11:36:59.432727 5166 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43103
I20260810 11:36:59.433089 5166 heartbeater.cc:464] Registering TS with master...
I20260810 11:36:59.433912 5166 heartbeater.cc:511] Master 127.3.240.254:43103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:36:59.436447 4902 ts_manager.cc:194] Registered new tserver with Master: 3be4cef8712e4687a6227b5633f0c88a (127.3.240.193:35949)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260810 11:36:59.438174 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:55257
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:36:59.467839 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:36:59.835397 5176 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:36:59.835688 5176 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:36:59.835901 5176 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:36:59.843847 5176 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:36:59.843962 5176 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:36:59.844137 5176 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:36:59.854259 5176 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:36:59.855911 5176 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:36:59.858093 5176 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:36:59.873255 5186 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:59.873193 5183 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:36:59.873299 5182 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:36:59.874698 5176 server_base.cc:1034] running on GCE node
I20260810 11:36:59.875381 5176 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:36:59.876318 5176 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:36:59.877599 5176 hybrid_clock.cc:648] HybridClock initialized: now 1786361819877473 us; error 66 us; skew 500 ppm
I20260810 11:36:59.877902 5176 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:36:59 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:36:59.881387 5176 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:36:59.883314 5176 webserver.cc:460] Webserver started at http://127.3.240.194:33179/ using document root <none> and password file <none>
I20260810 11:36:59.883978 5176 fs_manager.cc:359] Metadata directory not provided
I20260810 11:36:59.884079 5176 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:36:59.884403 5176 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:36:59.887648 5176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "95ba59246be4459683e107689ae9febb"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "b1045b366b48c88e5693554e9bdf98a7"
server_key_iv: "fe64894a7d4d61ab8e24fefea9665e00"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.888480 5176 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "95ba59246be4459683e107689ae9febb"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "b1045b366b48c88e5693554e9bdf98a7"
server_key_iv: "fe64894a7d4d61ab8e24fefea9665e00"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.893910 5176 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 11:36:59.898043 5196 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.899720 5176 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:36:59.899893 5176 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "95ba59246be4459683e107689ae9febb"
format_stamp: "Formatted at 2026-08-10 11:36:59 on dist-test-slave-drl0"
server_key: "b1045b366b48c88e5693554e9bdf98a7"
server_key_iv: "fe64894a7d4d61ab8e24fefea9665e00"
server_key_version: "encryptionkey@0"
I20260810 11:36:59.900104 5176 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:36:59.921586 5176 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:36:59.926203 5176 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:36:59.926535 5176 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:36:59.927687 5176 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:36:59.929054 5176 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:36:59.929131 5176 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.929219 5176 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:36:59.929266 5176 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:36:59.971546 5176 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:44587
I20260810 11:36:59.971583 5368 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:44587 every 8 connection(s)
I20260810 11:36:59.972913 5176 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:36:59.976528 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5176
I20260810 11:36:59.976784 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:36:59.977438 4035 external_mini_cluster.cc:1428] Setting key 9b2e711c4162e2a47cb97f64b1f5b28d
Aug 10 11:36:59 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361819, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:36:59.997753 5373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:59.976018 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54221 (local address 127.3.240.254:43103)
0810 11:36:59.976510 (+ 492us) server_negotiation.cc:206] Beginning negotiation
0810 11:36:59.976516 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:36:59.978174 (+ 1658us) server_negotiation.cc:407] Connection header received
0810 11:36:59.980612 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:36:59.980617 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:36:59.980688 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:36:59.980792 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:36:59.983395 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.984164 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.986760 (+ 2596us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.987300 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.987410 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:59.991841 (+ 4431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:36:59.991849 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:36:59.991856 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:36:59.991905 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:36:59.994622 (+ 2717us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:59.995453 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:59.995460 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:59.995468 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:59.995591 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:36:59.996019 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:36:59.996024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:36:59.996029 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:36:59.996292 (+ 263us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:36:59.996449 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 11:36:59.997133 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 11:36:59.997351 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":282,"threads_started":1}
I20260810 11:36:59.998885 5375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:36:59.976896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43103 (local address 127.3.240.194:54221)
0810 11:36:59.977992 (+ 1096us) negotiation.cc:107] Waiting for socket to connect
0810 11:36:59.978043 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 11:36:59.980073 (+ 2030us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:36:59.981124 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:36:59.981136 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:36:59.981955 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:36:59.983110 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:59.983156 (+ 46us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.984376 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.984386 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:59.986566 (+ 2180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:36:59.986586 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:36:59.987544 (+ 958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:36:59.987554 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:36:59.988173 (+ 619us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:36:59.989005 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:36:59.989046 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:36:59.991580 (+ 2534us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:36:59.994784 (+ 3204us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:59.994795 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:59.994812 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:59.995319 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:59.995707 (+ 388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:36:59.995711 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:36:59.995715 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:36:59.995897 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:36:59.996422 (+ 525us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:36:59.996430 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:36:59.996774 (+ 344us) client_negotiation.cc:769] Sending connection context
0810 11:36:59.997044 (+ 270us) client_negotiation.cc:240] Negotiation successful
0810 11:36:59.997585 (+ 541us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":821,"thread_start_us":520,"threads_started":1}
I20260810 11:37:00.001739 5371 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43103
I20260810 11:37:00.002146 5371 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:00.003029 5371 heartbeater.cc:511] Master 127.3.240.254:43103 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260810 11:37:00.006501 4902 ts_manager.cc:194] Registered new tserver with Master: 95ba59246be4459683e107689ae9febb (127.3.240.194:44587)
I20260810 11:37:00.007627 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:54221
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:00.038448 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:00.422857 5378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:00.423420 5378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:00.423615 5378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:00.434203 5378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:00.434335 5378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:00.434532 5378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:00.441982 5166 heartbeater.cc:503] Master 127.3.240.254:43103 was elected leader, sending a full tablet report...
I20260810 11:37:00.447036 5378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43103
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:00.448488 5378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:00.449970 5378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:00.464430 5385 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:00.464344 5384 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:00.464323 5388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:00.470134 5378 server_base.cc:1034] running on GCE node
I20260810 11:37:00.472761 5378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:00.473986 5378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:00.475265 5378 hybrid_clock.cc:648] HybridClock initialized: now 1786361820475220 us; error 35 us; skew 500 ppm
I20260810 11:37:00.475626 5378 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:00 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:00.479262 5378 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:00.480897 5378 webserver.cc:460] Webserver started at http://127.3.240.195:36213/ using document root <none> and password file <none>
I20260810 11:37:00.481666 5378 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:00.481771 5378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:00.482129 5378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:00.485608 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "c63dbefbf02a41daab58cfb6d7acccea"
format_stamp: "Formatted at 2026-08-10 11:37:00 on dist-test-slave-drl0"
server_key: "a1264dba31b992139deb0b5aa84559ad"
server_key_iv: "8a0dc3c99b79262175a2015e65675faa"
server_key_version: "encryptionkey@0"
I20260810 11:37:00.486454 5378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "c63dbefbf02a41daab58cfb6d7acccea"
format_stamp: "Formatted at 2026-08-10 11:37:00 on dist-test-slave-drl0"
server_key: "a1264dba31b992139deb0b5aa84559ad"
server_key_iv: "8a0dc3c99b79262175a2015e65675faa"
server_key_version: "encryptionkey@0"
I20260810 11:37:00.495213 5378 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.011s sys 0.000s
I20260810 11:37:00.499071 5396 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:00.500665 5378 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:00.500869 5378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "c63dbefbf02a41daab58cfb6d7acccea"
format_stamp: "Formatted at 2026-08-10 11:37:00 on dist-test-slave-drl0"
server_key: "a1264dba31b992139deb0b5aa84559ad"
server_key_iv: "8a0dc3c99b79262175a2015e65675faa"
server_key_version: "encryptionkey@0"
I20260810 11:37:00.501091 5378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:00.533666 5378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:00.538287 5378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:00.538614 5378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:00.539839 5378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:00.541380 5378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:00.541455 5378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:00.541589 5378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:00.541666 5378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:00.605111 5378 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34115
I20260810 11:37:00.605166 5561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34115 every 8 connection(s)
I20260810 11:37:00.606578 5378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:00.614312 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5378
I20260810 11:37:00.614550 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:00.615221 4035 external_mini_cluster.cc:1428] Setting key 8b0c67901b93b839b7c12170826f7387
Aug 10 11:37:00 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361820, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:00.635190 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.610431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35375 (local address 127.3.240.254:43103)
0810 11:37:00.612096 (+ 1665us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.612102 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.612285 (+ 183us) server_negotiation.cc:407] Connection header received
0810 11:37:00.615430 (+ 3145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.615437 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.615537 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.615685 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:00.618013 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.619310 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.621463 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.622040 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.622168 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.626159 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:00.626165 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:00.626168 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:00.626201 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:00.630792 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:00.631986 (+ 1194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:00.631995 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:00.632000 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:00.632208 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:00.632712 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:00.632716 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:00.632718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:00.633019 (+ 301us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:00.633151 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.634351 (+ 1200us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.634687 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1510,"thread_start_us":418,"threads_started":1}
I20260810 11:37:00.636082 5566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.610914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43103 (local address 127.3.240.195:35375)
0810 11:37:00.612064 (+ 1150us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.612114 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.614981 (+ 2867us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.616002 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.616011 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.616907 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:00.617767 (+ 860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.617803 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.619613 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.619622 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.621252 (+ 1630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.621268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.622305 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.622311 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.622571 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.623464 (+ 893us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:00.623488 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:00.625935 (+ 2447us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:00.631082 (+ 5147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:00.631090 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:00.631104 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:00.631824 (+ 720us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:00.632348 (+ 524us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:00.632352 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:00.632354 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:00.632570 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:00.633190 (+ 620us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:00.633197 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:00.633618 (+ 421us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.633984 (+ 366us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.634512 (+ 528us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":796,"thread_start_us":517,"threads_started":1}
I20260810 11:37:00.638156 5562 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43103
I20260810 11:37:00.638571 5562 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:00.639873 5562 heartbeater.cc:511] Master 127.3.240.254:43103 requested a full tablet report, sending...
I20260810 11:37:00.641849 4902 ts_manager.cc:194] Registered new tserver with Master: c63dbefbf02a41daab58cfb6d7acccea (127.3.240.195:34115)
I20260810 11:37:00.643958 4902 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:35375
I20260810 11:37:00.646641 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:37:00.677551 5565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.656193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58530 (local address 127.3.240.254:43103)
0810 11:37:00.656534 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.656540 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.657791 (+ 1251us) server_negotiation.cc:407] Connection header received
0810 11:37:00.658022 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.658029 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.658369 (+ 340us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.658498 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:00.661836 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.662813 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.664586 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.665048 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.665189 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.668634 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:00.668639 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:00.668642 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:00.668675 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:00.673242 (+ 4567us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:00.674394 (+ 1152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:00.674400 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:00.674402 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:00.674648 (+ 246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:00.675353 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:00.675357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:00.675359 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:00.675828 (+ 469us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:00.676139 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.676732 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.677009 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136}
I20260810 11:37:00.690677 4902 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:00.702539 4902 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:00.754125 5581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.723796 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44587 (local address 127.0.0.1:45492)
0810 11:37:00.736225 (+ 12429us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.736310 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.738363 (+ 2053us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.742872 (+ 4509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.742878 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.742908 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:00.743836 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.743856 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.746773 (+ 2917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.746788 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.749546 (+ 2758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.749566 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.752662 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.752675 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.752917 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.753080 (+ 163us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.753261 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.753498 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11954,"mutex_wait_us":74,"thread_start_us":17702,"threads_started":1}
I20260810 11:37:00.754365 5579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.723964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45492 (local address 127.3.240.194:44587)
0810 11:37:00.741006 (+ 17042us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.741021 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.741064 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:37:00.741197 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.741209 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.741542 (+ 333us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.741909 (+ 367us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:00.744095 (+ 2186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.746534 (+ 2439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.749787 (+ 3253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.751884 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.752033 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.752236 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.753424 (+ 1188us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.753682 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16749,"thread_start_us":16782,"threads_started":1}
I20260810 11:37:00.758489 5585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.737680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35949 (local address 127.0.0.1:45414)
0810 11:37:00.741145 (+ 3465us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.741172 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.741356 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.745611 (+ 4255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.745615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.745645 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:00.747201 (+ 1556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.747228 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.751200 (+ 3972us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.751212 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.755380 (+ 4168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.755394 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.757548 (+ 2154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.757557 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.757744 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.757781 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.757887 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.758024 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2900,"mutex_wait_us":50,"thread_start_us":2661,"threads_started":1}
I20260810 11:37:00.758661 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.738144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45414 (local address 127.3.240.193:35949)
0810 11:37:00.744732 (+ 6588us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.744752 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.744780 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:37:00.744892 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.744900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.745283 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.745619 (+ 336us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:00.747537 (+ 1918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.750922 (+ 3385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.755555 (+ 4633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.757296 (+ 1741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.757468 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.757734 (+ 266us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.757948 (+ 214us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.758114 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6339,"thread_start_us":5979,"threads_started":1}
I20260810 11:37:00.760342 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.724124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34115 (local address 127.0.0.1:54994)
0810 11:37:00.737156 (+ 13032us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.737175 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.743951 (+ 6776us) spinlock_profiling.cc:243] Waited 5.67 ms on lock 0x61200002c928. stack: 00007fb73d96a66e 00007fb73d96a430 00007fb73c9f15ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007fb73d9600fd 00007fb73d95fd1e 00007fb7508693cd 00007fb75086936c 00007fb750835b3b 00007fb74436f536 00007fb74436dd9e 00007fb7443f5dca 00007fb7443f3f68 00007fb74442782f
0810 11:37:00.744011 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.744944 (+ 933us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.744949 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.744975 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:00.746400 (+ 1425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.746438 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.751221 (+ 4783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.751228 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.755711 (+ 4483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.755729 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.759035 (+ 3306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.759043 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.759358 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.759394 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.759501 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.759635 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12885,"mutex_wait_us":90,"spinlock_wait_cycles":13046656,"thread_start_us":12390,"threads_started":1}
I20260810 11:37:00.761386 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.724427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54994 (local address 127.3.240.195:34115)
0810 11:37:00.743343 (+ 18916us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.743355 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.743443 (+ 88us) server_negotiation.cc:407] Connection header received
0810 11:37:00.744349 (+ 906us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.744366 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.744664 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.744914 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:00.747044 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.750598 (+ 3554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.755979 (+ 5381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.758444 (+ 2465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.758637 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.758909 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.760597 (+ 1688us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.760805 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18529,"thread_start_us":17971,"threads_started":1}
I20260810 11:37:00.761116 5267 tablet_service.cc:1467] Processing CreateTablet for tablet 5e68cec187ae4af58968f1bd5e7d5392 (DEFAULT_TABLE table=test-table [id=a52c1e709f5149b4bb5dcf309c732b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:00.763008 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e68cec187ae4af58968f1bd5e7d5392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:00.762501 5072 tablet_service.cc:1467] Processing CreateTablet for tablet 5e68cec187ae4af58968f1bd5e7d5392 (DEFAULT_TABLE table=test-table [id=a52c1e709f5149b4bb5dcf309c732b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:00.764039 5072 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e68cec187ae4af58968f1bd5e7d5392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:00.775439 5461 tablet_service.cc:1467] Processing CreateTablet for tablet 5e68cec187ae4af58968f1bd5e7d5392 (DEFAULT_TABLE table=test-table [id=a52c1e709f5149b4bb5dcf309c732b97]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:00.777318 5461 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5e68cec187ae4af58968f1bd5e7d5392. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:00.780375 5589 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Bootstrap starting.
I20260810 11:37:00.785588 5589 tablet_bootstrap.cc:654] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:00.787012 5591 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Bootstrap starting.
I20260810 11:37:00.787850 5589 log.cc:824] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:00.793366 5591 tablet_bootstrap.cc:654] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:00.796034 5589 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: No bootstrap required, opened a new log
I20260810 11:37:00.795980 5591 log.cc:824] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:00.796511 5589 ts_tablet_manager.cc:1397] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Time spent bootstrapping tablet: real 0.016s user 0.013s sys 0.000s
I20260810 11:37:00.800236 5594 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Bootstrap starting.
I20260810 11:37:00.801684 5591 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: No bootstrap required, opened a new log
I20260810 11:37:00.802136 5591 ts_tablet_manager.cc:1397] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Time spent bootstrapping tablet: real 0.015s user 0.008s sys 0.004s
I20260810 11:37:00.806696 5594 tablet_bootstrap.cc:654] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:00.808337 5594 log.cc:824] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:00.812458 5594 tablet_bootstrap.cc:492] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: No bootstrap required, opened a new log
I20260810 11:37:00.812839 5594 ts_tablet_manager.cc:1397] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260810 11:37:00.814493 5589 raft_consensus.cc:348] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.815232 5589 raft_consensus.cc:374] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:00.815408 5589 raft_consensus.cc:729] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3be4cef8712e4687a6227b5633f0c88a, State: Initialized, Role: FOLLOWER
I20260810 11:37:00.816797 5589 consensus_queue.cc:260] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.818907 5591 raft_consensus.cc:348] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.819896 5591 raft_consensus.cc:374] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:00.820258 5591 raft_consensus.cc:729] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ba59246be4459683e107689ae9febb, State: Initialized, Role: FOLLOWER
I20260810 11:37:00.820415 5589 ts_tablet_manager.cc:1428] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Time spent starting tablet: real 0.024s user 0.016s sys 0.008s
I20260810 11:37:00.821307 5591 consensus_queue.cc:260] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.823967 5371 heartbeater.cc:503] Master 127.3.240.254:43103 was elected leader, sending a full tablet report...
I20260810 11:37:00.824570 5591 ts_tablet_manager.cc:1428] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Time spent starting tablet: real 0.022s user 0.008s sys 0.015s
I20260810 11:37:00.826108 5594 raft_consensus.cc:348] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.826700 5594 raft_consensus.cc:374] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:00.826891 5594 raft_consensus.cc:729] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63dbefbf02a41daab58cfb6d7acccea, State: Initialized, Role: FOLLOWER
I20260810 11:37:00.827620 5594 consensus_queue.cc:260] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.829841 5562 heartbeater.cc:503] Master 127.3.240.254:43103 was elected leader, sending a full tablet report...
I20260810 11:37:00.829948 5594 ts_tablet_manager.cc:1428] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Time spent starting tablet: real 0.017s user 0.018s sys 0.000s
W20260810 11:37:00.859234 5563 tablet.cc:2369] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:00.905031 5167 tablet.cc:2369] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:00.925918 5597 raft_consensus.cc:482] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:00.926502 5597 raft_consensus.cc:504] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.929849 5597 leader_election.cc:288] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c63dbefbf02a41daab58cfb6d7acccea (127.3.240.195:34115), 3be4cef8712e4687a6227b5633f0c88a (127.3.240.193:35949)
I20260810 11:37:00.942449 5582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.930585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53307 (local address 127.3.240.195:34115)
0810 11:37:00.930909 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.930927 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.932326 (+ 1399us) server_negotiation.cc:407] Connection header received
0810 11:37:00.932697 (+ 371us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.932704 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.932787 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.932855 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:00.934456 (+ 1601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.936332 (+ 1876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.939444 (+ 3112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.940598 (+ 1154us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.940827 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.940979 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.941612 (+ 633us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.941827 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 11:37:00.942907 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.933736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54321 (local address 127.3.240.193:35949)
0810 11:37:00.934063 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.934070 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.934093 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:00.934183 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.934188 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.934259 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.934349 (+ 90us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:00.935309 (+ 960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.936846 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.940171 (+ 3325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.941412 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.941629 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.941772 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:00.942385 (+ 613us) server_negotiation.cc:299] Negotiation successful
0810 11:37:00.942528 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260810 11:37:00.942986 5602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.930201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34115 (local address 127.3.240.194:53307)
0810 11:37:00.932186 (+ 1985us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.932218 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.932500 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.933205 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.933212 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.933257 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:00.934265 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.934282 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.936511 (+ 2229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.936515 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.939283 (+ 2768us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.939296 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.940862 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.940870 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.941259 (+ 389us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.941319 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.941544 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.941858 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1718,"mutex_wait_us":73,"thread_start_us":1144,"threads_started":1}
I20260810 11:37:00.943444 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.931955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35949 (local address 127.3.240.194:54321)
0810 11:37:00.933134 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:00.933151 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:00.933356 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:00.934522 (+ 1166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:00.934527 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:00.934556 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:00.935075 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.935093 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.937014 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.937019 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.939994 (+ 2975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:00.940009 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.941817 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.941827 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:00.942189 (+ 362us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:00.942230 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 11:37:00.942358 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 11:37:00.942523 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":26,"spinlock_wait_cycles":25856,"thread_start_us":731,"threads_started":1}
I20260810 11:37:00.944553 5489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e68cec187ae4af58968f1bd5e7d5392" candidate_uuid: "95ba59246be4459683e107689ae9febb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63dbefbf02a41daab58cfb6d7acccea" is_pre_election: true
I20260810 11:37:00.945262 5489 raft_consensus.cc:2454] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ba59246be4459683e107689ae9febb in term 0.
I20260810 11:37:00.946149 5106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e68cec187ae4af58968f1bd5e7d5392" candidate_uuid: "95ba59246be4459683e107689ae9febb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be4cef8712e4687a6227b5633f0c88a" is_pre_election: true
I20260810 11:37:00.946502 5197 leader_election.cc:302] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95ba59246be4459683e107689ae9febb, c63dbefbf02a41daab58cfb6d7acccea; no voters:
I20260810 11:37:00.946861 5106 raft_consensus.cc:2454] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 95ba59246be4459683e107689ae9febb in term 0.
I20260810 11:37:00.947113 5597 raft_consensus.cc:2781] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:00.947321 5597 raft_consensus.cc:482] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:00.947471 5597 raft_consensus.cc:3037] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:00.950465 5597 raft_consensus.cc:504] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.951287 5597 leader_election.cc:288] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [CANDIDATE]: Term 1 election: Requested vote from peers c63dbefbf02a41daab58cfb6d7acccea (127.3.240.195:34115), 3be4cef8712e4687a6227b5633f0c88a (127.3.240.193:35949)
I20260810 11:37:00.951697 5106 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e68cec187ae4af58968f1bd5e7d5392" candidate_uuid: "95ba59246be4459683e107689ae9febb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3be4cef8712e4687a6227b5633f0c88a"
I20260810 11:37:00.951694 5489 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5e68cec187ae4af58968f1bd5e7d5392" candidate_uuid: "95ba59246be4459683e107689ae9febb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c63dbefbf02a41daab58cfb6d7acccea"
I20260810 11:37:00.951944 5106 raft_consensus.cc:3037] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:00.951961 5489 raft_consensus.cc:3037] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:00.955415 5489 raft_consensus.cc:2454] T 5e68cec187ae4af58968f1bd5e7d5392 P c63dbefbf02a41daab58cfb6d7acccea [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ba59246be4459683e107689ae9febb in term 1.
I20260810 11:37:00.955662 5106 raft_consensus.cc:2454] T 5e68cec187ae4af58968f1bd5e7d5392 P 3be4cef8712e4687a6227b5633f0c88a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 95ba59246be4459683e107689ae9febb in term 1.
I20260810 11:37:00.955915 5197 leader_election.cc:302] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 95ba59246be4459683e107689ae9febb, c63dbefbf02a41daab58cfb6d7acccea; no voters:
I20260810 11:37:00.956506 5597 raft_consensus.cc:2781] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:00.957481 5597 raft_consensus.cc:686] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [term 1 LEADER]: Becoming Leader. State: Replica: 95ba59246be4459683e107689ae9febb, State: Running, Role: LEADER
I20260810 11:37:00.958312 5597 consensus_queue.cc:237] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } }
I20260810 11:37:00.965113 4901 catalog_manager.cc:5495] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb reported cstate change: term changed from 0 to 1, leader changed from <none> to 95ba59246be4459683e107689ae9febb (127.3.240.194). New cstate: current_term: 1 leader_uuid: "95ba59246be4459683e107689ae9febb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c63dbefbf02a41daab58cfb6d7acccea" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "95ba59246be4459683e107689ae9febb" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44587 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3be4cef8712e4687a6227b5633f0c88a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 35949 } health_report { overall_health: UNKNOWN } } }
W20260810 11:37:00.975328 5372 tablet.cc:2369] T 5e68cec187ae4af58968f1bd5e7d5392 P 95ba59246be4459683e107689ae9febb: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 11:37:00 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361818, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:01.011427 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:00.986529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45416 (local address 127.3.240.193:35949)
0810 11:37:00.986800 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:00.986810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:00.987421 (+ 611us) server_negotiation.cc:407] Connection header received
0810 11:37:00.987583 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:00.987589 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:00.987757 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:00.987865 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:00.990633 (+ 2768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.992513 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.994708 (+ 2195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:00.995564 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:00.995704 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.001987 (+ 6283us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.002007 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.002025 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.002073 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.007460 (+ 5387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.008511 (+ 1051us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.008520 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.008523 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.008722 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.009233 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.009240 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.009242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.009817 (+ 575us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.010130 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.010537 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.010911 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:01.081198 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.061729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45428 (local address 127.3.240.193:35949)
0810 11:37:01.062036 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:01.062046 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:01.062622 (+ 576us) server_negotiation.cc:407] Connection header received
0810 11:37:01.062916 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:01.062921 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:01.063063 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:01.063188 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:01.065474 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.066313 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.067926 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.068428 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.068581 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.073463 (+ 4882us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.073469 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.073473 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.073509 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.077776 (+ 4267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.078582 (+ 806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.078590 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.078592 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.078887 (+ 295us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.079664 (+ 777us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.079669 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.079672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.079998 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.080313 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.080583 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.080803 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:01.148602 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.125162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45432 (local address 127.3.240.193:35949)
0810 11:37:01.125595 (+ 433us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:01.125602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:01.125924 (+ 322us) server_negotiation.cc:407] Connection header received
0810 11:37:01.126155 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:01.126160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:01.126234 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:01.126352 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:01.128444 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.129369 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.131499 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.132160 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.132323 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.139458 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.139483 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.139490 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.139642 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.144338 (+ 4696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.145426 (+ 1088us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.145431 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.145433 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.145628 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.146170 (+ 542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.146174 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.146177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.147092 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.147527 (+ 435us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.147758 (+ 231us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.148126 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":213}
W20260810 11:37:01.151543 5044 tablet_service.cc:2908] Rejecting scan request for tablet 5e68cec187ae4af58968f1bd5e7d5392: Uninitialized: clean time has not yet been initialized
I20260810 11:37:01.158056 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4985
I20260810 11:37:01.176293 5586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.159705 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45438 (local address 127.3.240.193:35949)
0810 11:37:01.160050 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:01.160054 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:01.160072 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:01.160121 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:01.160125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:01.160198 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:01.160284 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:01.161679 (+ 1395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.162407 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.163838 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.164279 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.164399 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.168420 (+ 4021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.168425 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.168429 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.168462 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.172166 (+ 3704us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.173401 (+ 1235us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.173406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.173408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.173583 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.174154 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.174159 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.174161 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.174586 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.175003 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.175362 (+ 359us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.175575 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":223}
I20260810 11:37:01.177453 5159 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:01.274408 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4985
I20260810 11:37:01.300263 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5176
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:01.318778 5624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.301252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45496 (local address 127.3.240.194:44587)
0810 11:37:01.301764 (+ 512us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:01.301768 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:01.301794 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:37:01.301852 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:01.301856 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:01.301927 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:01.302035 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:01.303794 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.305049 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.306720 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.307554 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.307670 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.312010 (+ 4340us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.312024 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.312056 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.312095 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.315367 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.316459 (+ 1092us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.316466 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.316468 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.316618 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.317277 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.317282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.317284 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.317678 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.317931 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.318305 (+ 374us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.318505 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":304,"threads_started":1}
I20260810 11:37:01.319556 5362 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:01.419209 5603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.418079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:35949 (local address 127.3.240.194:39375)
0810 11:37:01.418290 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:01.418792 (+ 502us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35949: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
W20260810 11:37:01.419571 5197 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.193:35949: connect: Connection refused (error 111)
I20260810 11:37:01.420172 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5176
W20260810 11:37:01.451169 5403 connection.cc:535] server connection from 127.3.240.194:53307 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 11:37:01.451771 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5378
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[4823](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361821, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:01.469820 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:01.452545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54998 (local address 127.3.240.195:34115)
0810 11:37:01.453119 (+ 574us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:01.453123 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:01.453141 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:01.453200 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:01.453205 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:01.453278 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:01.453410 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:01.454792 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.455607 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.456957 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:01.457377 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:01.457531 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:01.462633 (+ 5102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:01.462647 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:01.462664 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:01.462714 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:01.466204 (+ 3490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.467322 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.467331 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.467333 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.467511 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:01.467965 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:01.467969 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:01.467971 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:01.468367 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:01.468604 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:01.469252 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 11:37:01.469521 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":286,"threads_started":1}
I20260810 11:37:01.470479 5555 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:01.566233 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5378
I20260810 11:37:01.588370 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4850
I20260810 11:37:01.589756 4941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:01.708793 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4850
2026-08-10T11:37:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (5129 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[5633](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[5633](info): set up 2 sockets
Aug 10 11:37:01 dist-test-slave-drl0 krb5kdc[5633](info): commencing operation
krb5kdc: starting...
W20260810 11:37:03.053117 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.290s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:03 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:03.292133 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37549
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37549
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:03.659528 5652 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:03.659850 5652 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:03.659958 5652 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:03.670157 5652 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:03.670286 5652 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 11:37:03.670336 5652 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:03.670399 5652 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:03.670464 5652 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:03.670542 5652 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:03.682633 5652 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37549
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37549
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:03.684357 5652 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:03.685886 5652 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:03.699199 5659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:03.699270 5663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:03.699246 5660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:03.699972 5652 server_base.cc:1034] running on GCE node
I20260810 11:37:03.700978 5652 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:03.702389 5652 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:03.703682 5652 hybrid_clock.cc:648] HybridClock initialized: now 1786361823703595 us; error 48 us; skew 500 ppm
I20260810 11:37:03.704046 5652 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:03 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:03.711395 5652 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:03.713274 5652 webserver.cc:460] Webserver started at http://127.3.240.254:34955/ using document root <none> and password file <none>
I20260810 11:37:03.714066 5652 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:03.714228 5652 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:03.714620 5652 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:03.719259 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "0d7117160b044926acab7671647c1c4d"
format_stamp: "Formatted at 2026-08-10 11:37:03 on dist-test-slave-drl0"
server_key: "92f4a49a98e9e301aea0ae916e1797b7"
server_key_iv: "30a30b9aaf1230b7ea8a049b6adb94c8"
server_key_version: "encryptionkey@0"
I20260810 11:37:03.720330 5652 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "0d7117160b044926acab7671647c1c4d"
format_stamp: "Formatted at 2026-08-10 11:37:03 on dist-test-slave-drl0"
server_key: "92f4a49a98e9e301aea0ae916e1797b7"
server_key_iv: "30a30b9aaf1230b7ea8a049b6adb94c8"
server_key_version: "encryptionkey@0"
I20260810 11:37:03.727823 5652 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 11:37:03.732277 5673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:03.734417 5652 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 11:37:03.734717 5652 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "0d7117160b044926acab7671647c1c4d"
format_stamp: "Formatted at 2026-08-10 11:37:03 on dist-test-slave-drl0"
server_key: "92f4a49a98e9e301aea0ae916e1797b7"
server_key_iv: "30a30b9aaf1230b7ea8a049b6adb94c8"
server_key_version: "encryptionkey@0"
I20260810 11:37:03.735047 5652 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:03.753739 5652 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:03.758514 5652 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:03.759109 5652 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:03.786757 5652 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37549
I20260810 11:37:03.786801 5757 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37549 every 8 connection(s)
I20260810 11:37:03.788403 5652 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:03.792626 5759 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:03.797850 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5652
I20260810 11:37:03.798271 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:03.799218 4035 external_mini_cluster.cc:1428] Setting key b8de8eb0b2c3c92b848a84bb443dbd9d
I20260810 11:37:03.804193 5759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Bootstrap starting.
I20260810 11:37:03.808090 5759 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:03.809258 5759 log.cc:824] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:03.812502 5759 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: No bootstrap required, opened a new log
Aug 10 11:37:03 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361823, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:03.822108 5759 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } }
I20260810 11:37:03.822577 5759 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:03.822712 5759 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d7117160b044926acab7671647c1c4d, State: Initialized, Role: FOLLOWER
I20260810 11:37:03.823439 5759 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } }
I20260810 11:37:03.823734 5759 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:03.823947 5759 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:03.824168 5759 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:03.825534 5762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:03.801462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37602 (local address 127.3.240.254:37549)
0810 11:37:03.802366 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:03.802377 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:03.802418 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:37:03.804161 (+ 1743us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:03.804187 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:03.804609 (+ 422us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:03.804923 (+ 314us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:03.807142 (+ 2219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:03.808886 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:03.810536 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:03.811261 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:03.811398 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:03.817112 (+ 5714us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:03.817120 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:03.817135 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:03.817179 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:03.820858 (+ 3679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:03.821731 (+ 873us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:03.821738 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:03.821745 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:03.821913 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:03.822385 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:03.822389 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:03.822391 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:03.823243 (+ 852us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:03.823475 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:03.824092 (+ 617us) server_negotiation.cc:299] Negotiation successful
0810 11:37:03.824513 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":446,"threads_started":1}
I20260810 11:37:03.826639 5759 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } }
I20260810 11:37:03.827301 5759 leader_election.cc:302] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0d7117160b044926acab7671647c1c4d; no voters:
I20260810 11:37:03.827979 5759 leader_election.cc:288] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:03.828189 5770 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:03.829221 5770 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [term 1 LEADER]: Becoming Leader. State: Replica: 0d7117160b044926acab7671647c1c4d, State: Running, Role: LEADER
I20260810 11:37:03.829792 5770 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } }
I20260810 11:37:03.830369 5759 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:03.833281 5771 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0d7117160b044926acab7671647c1c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } } }
I20260810 11:37:03.833516 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d7117160b044926acab7671647c1c4d. Latest consensus state: current_term: 1 leader_uuid: "0d7117160b044926acab7671647c1c4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0d7117160b044926acab7671647c1c4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37549 } } }
I20260810 11:37:03.833868 5771 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:03.833959 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:03.835510 5784 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:03.843386 5784 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:03.852860 5784 catalog_manager.cc:1347] Generated new cluster ID: 58a14cbb033244a681b86fa0b7b27b81
I20260810 11:37:03.852998 5784 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:03.873942 5784 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:03.876278 5784 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:03.888649 5784 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Generated new TSK 0
I20260810 11:37:03.889726 5784 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:03.959112 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37549
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:04.365221 5800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:04.365567 5800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:04.365749 5800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:04.373545 5800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:04.373808 5800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:04.373999 5800 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:04.374146 5800 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:04.385860 5800 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:04.387324 5800 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:04.388875 5800 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:04.403280 5809 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:04.403411 5805 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:04.403373 5807 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:04.404539 5800 server_base.cc:1034] running on GCE node
I20260810 11:37:04.405273 5800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:04.406483 5800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:04.407914 5800 hybrid_clock.cc:648] HybridClock initialized: now 1786361824407777 us; error 51 us; skew 500 ppm
I20260810 11:37:04.408490 5800 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:04 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:04.412771 5800 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:04.414880 5800 webserver.cc:460] Webserver started at http://127.3.240.193:37607/ using document root <none> and password file <none>
I20260810 11:37:04.415678 5800 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:04.415823 5800 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:04.416167 5800 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:04.419940 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "a39022873c814a00bed1404afeda2505"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "4ed071b61dec2c490a9d60fb67263cdb"
server_key_iv: "e0a5f83792b7a39f5da89faaf1cd0512"
server_key_version: "encryptionkey@0"
I20260810 11:37:04.421111 5800 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "a39022873c814a00bed1404afeda2505"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "4ed071b61dec2c490a9d60fb67263cdb"
server_key_iv: "e0a5f83792b7a39f5da89faaf1cd0512"
server_key_version: "encryptionkey@0"
I20260810 11:37:04.428093 5800 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 11:37:04.432572 5821 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:04.434518 5800 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:37:04.434728 5800 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "a39022873c814a00bed1404afeda2505"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "4ed071b61dec2c490a9d60fb67263cdb"
server_key_iv: "e0a5f83792b7a39f5da89faaf1cd0512"
server_key_version: "encryptionkey@0"
I20260810 11:37:04.435111 5800 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:04.450822 5800 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:04.456652 5800 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:04.456990 5800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:04.460078 5800 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:04.460191 5800 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:04.460371 5800 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:04.460465 5800 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:04 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:04.486338 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:04.460608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49625 (local address 127.3.240.254:37549)
0810 11:37:04.461437 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:04.461443 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:04.462473 (+ 1030us) server_negotiation.cc:407] Connection header received
0810 11:37:04.464839 (+ 2366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:04.464845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:04.464952 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:04.465110 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:04.467640 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.469029 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.471856 (+ 2827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.472296 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.472446 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:04.476579 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:04.476585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:04.476589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:04.476653 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:04.481274 (+ 4621us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:04.482562 (+ 1288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:04.482570 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:04.482573 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:04.482792 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:04.483491 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:04.483507 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:04.483513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:04.483888 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:04.484033 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:04.485273 (+ 1240us) server_negotiation.cc:299] Negotiation successful
0810 11:37:04.485759 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":560,"threads_started":1}
I20260810 11:37:04.488260 5835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:04.461303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.193:49625)
0810 11:37:04.462290 (+ 987us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:04.462340 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:04.464541 (+ 2201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:04.465439 (+ 898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:04.465455 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:04.466231 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:04.467385 (+ 1154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:04.467419 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.469220 (+ 1801us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.469234 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:04.471670 (+ 2436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:04.471688 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.472571 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.472580 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:04.472809 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:04.473560 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:04.473585 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:04.476240 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:04.481452 (+ 5212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:04.481469 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:04.481491 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:04.482302 (+ 811us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:04.482980 (+ 678us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:04.482988 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:04.482993 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:04.483314 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:04.484046 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:04.484057 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:04.484649 (+ 592us) client_negotiation.cc:769] Sending connection context
0810 11:37:04.485066 (+ 417us) client_negotiation.cc:240] Negotiation successful
0810 11:37:04.486169 (+ 1103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":430,"threads_started":1}
I20260810 11:37:04.515339 5834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:04.497322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51303 (local address 127.3.240.254:37549)
0810 11:37:04.497557 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:04.497565 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:04.498290 (+ 725us) server_negotiation.cc:407] Connection header received
0810 11:37:04.498610 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:04.498617 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:04.498712 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:04.498829 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:04.500124 (+ 1295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.501155 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.503471 (+ 2316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.504038 (+ 567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.504222 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:04.508172 (+ 3950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:04.508177 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:04.508181 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:04.508213 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:04.512454 (+ 4241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:04.513293 (+ 839us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:04.513299 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:04.513304 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:04.513454 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:04.513843 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:04.513848 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:04.513850 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:04.514133 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:04.514306 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:04.514715 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 11:37:04.514972 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 11:37:04.515547 5925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:04.497349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.193:51303)
0810 11:37:04.498155 (+ 806us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:04.498180 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:04.498361 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:04.498868 (+ 507us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:04.498873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:04.499477 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:04.499967 (+ 490us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:04.499979 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.501317 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.501324 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:04.503300 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:04.503314 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:04.504376 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:04.504389 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:04.504851 (+ 462us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:04.505639 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:04.505669 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:04.507953 (+ 2284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:04.512611 (+ 4658us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:04.512617 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:04.512621 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:04.513148 (+ 527us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:04.513575 (+ 427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:04.513581 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:04.513584 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:04.513716 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:04.514261 (+ 545us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:04.514266 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:04.514453 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 11:37:04.514593 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 11:37:04.514972 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":597,"thread_start_us":386,"threads_started":1}
I20260810 11:37:04.523552 5800 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44885
I20260810 11:37:04.523577 5992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44885 every 8 connection(s)
I20260810 11:37:04.524799 5800 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:04.527133 5993 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37549
I20260810 11:37:04.527567 5993 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:04.528517 5993 heartbeater.cc:511] Master 127.3.240.254:37549 requested a full tablet report, sending...
I20260810 11:37:04.529770 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5800
I20260810 11:37:04.530066 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:04.530829 4035 external_mini_cluster.cc:1428] Setting key 64fa5b9c37c6066320b74ad14d0c16f1
I20260810 11:37:04.532066 5697 ts_manager.cc:194] Registered new tserver with Master: a39022873c814a00bed1404afeda2505 (127.3.240.193:44885)
I20260810 11:37:04.534142 5697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:49625
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:04.587276 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37549
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:04.951817 6001 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:04.952116 6001 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:04.952302 6001 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:04.959872 6001 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:04.959997 6001 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:04.960146 6001 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:04.960274 6001 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:04.970639 6001 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:04.972014 6001 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:04.973491 6001 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:04.988715 6007 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:04.988701 6011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:04.988665 6009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:04.989596 6001 server_base.cc:1034] running on GCE node
I20260810 11:37:04.990561 6001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:04.992079 6001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:04.993453 6001 hybrid_clock.cc:648] HybridClock initialized: now 1786361824993363 us; error 41 us; skew 500 ppm
I20260810 11:37:04.993803 6001 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:04 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:04.997552 6001 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:04.999759 6001 webserver.cc:460] Webserver started at http://127.3.240.194:46625/ using document root <none> and password file <none>
I20260810 11:37:05.000557 6001 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:05.000706 6001 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:05.001082 6001 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:05.004408 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "6daa2b29536140e8b7ede072e722b862"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "68195a6d29f976a2407531aafacfa3d0"
server_key_iv: "219cee3be075a6184758220e78103273"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.005311 6001 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "6daa2b29536140e8b7ede072e722b862"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "68195a6d29f976a2407531aafacfa3d0"
server_key_iv: "219cee3be075a6184758220e78103273"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.010812 6001 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 11:37:05.014844 6020 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:05.016821 6001 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:05.017012 6001 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "6daa2b29536140e8b7ede072e722b862"
format_stamp: "Formatted at 2026-08-10 11:37:04 on dist-test-slave-drl0"
server_key: "68195a6d29f976a2407531aafacfa3d0"
server_key_iv: "219cee3be075a6184758220e78103273"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.017236 6001 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:05.032966 6001 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:05.037335 6001 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:05.037611 6001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:05.041548 6001 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:05.041674 6001 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:05.041844 6001 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:05.041960 6001 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:05 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361824, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:05.075476 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.043017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:55833 (local address 127.3.240.254:37549)
0810 11:37:05.043676 (+ 659us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.043685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.045572 (+ 1887us) server_negotiation.cc:407] Connection header received
0810 11:37:05.048762 (+ 3190us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.048771 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.048878 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.049041 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:05.050726 (+ 1685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.052406 (+ 1680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.058082 (+ 5676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.058614 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.058799 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.065121 (+ 6322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:05.065131 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:05.065140 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:05.065198 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:05.070693 (+ 5495us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.071886 (+ 1193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.071893 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.071897 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.072101 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.072852 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.072860 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.072866 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.073265 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:05.073472 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.074653 (+ 1181us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.074977 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":365,"threads_started":1}
I20260810 11:37:05.078151 6031 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.044326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.194:55833)
0810 11:37:05.045337 (+ 1011us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.045402 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.048363 (+ 2961us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.049326 (+ 963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.049334 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.049855 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:05.050447 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.050466 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.052673 (+ 2207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.052681 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.057860 (+ 5179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.057889 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.058912 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.058919 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.059276 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.060239 (+ 963us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:05.060278 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:05.064554 (+ 4276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:05.070886 (+ 6332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.070905 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.070924 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.071643 (+ 719us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.072252 (+ 609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.072267 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.072273 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.072620 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.073455 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:05.073463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:05.074074 (+ 611us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.074502 (+ 428us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.075736 (+ 1234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":453,"threads_started":1}
I20260810 11:37:05.103503 6001 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:40359
I20260810 11:37:05.103508 6193 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:40359 every 8 connection(s)
I20260810 11:37:05.103960 6030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.087505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:36857 (local address 127.3.240.254:37549)
0810 11:37:05.087831 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.087843 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.088623 (+ 780us) server_negotiation.cc:407] Connection header received
0810 11:37:05.088857 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.088862 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.088932 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.089026 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:05.090552 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.092032 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.093856 (+ 1824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.094659 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.094806 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.097617 (+ 2811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:05.097623 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:05.097628 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:05.097699 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:05.101173 (+ 3474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.101865 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.101873 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.101878 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.102006 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.102397 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.102403 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.102406 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.102741 (+ 335us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:05.102908 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.103392 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.103621 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260810 11:37:05.104125 6155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.087485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.194:36857)
0810 11:37:05.088471 (+ 986us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.088499 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.088694 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.089097 (+ 403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.089105 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.089873 (+ 768us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:05.090370 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.090384 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.092288 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.092297 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.093681 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.093696 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.095109 (+ 1413us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.095120 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.095306 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.096138 (+ 832us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:05.096156 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:05.097421 (+ 1265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:05.101325 (+ 3904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.101331 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.101335 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.101733 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.102128 (+ 395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.102133 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.102137 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.102277 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.102881 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:05.102887 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:05.103105 (+ 218us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.103267 (+ 162us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.103671 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":773,"thread_start_us":477,"threads_started":1}
I20260810 11:37:05.105263 6001 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:05.108397 6194 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37549
I20260810 11:37:05.108680 6194 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:05.109480 6194 heartbeater.cc:511] Master 127.3.240.254:37549 requested a full tablet report, sending...
I20260810 11:37:05.110905 5697 ts_manager.cc:194] Registered new tserver with Master: 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359)
I20260810 11:37:05.111718 5697 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:55833
I20260810 11:37:05.115334 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 6001
I20260810 11:37:05.115624 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:05.116190 4035 external_mini_cluster.cc:1428] Setting key 4233704703d35c886a5f1b80d0e589fa
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:05.180145 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:37549
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34351
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:05.537962 5993 heartbeater.cc:503] Master 127.3.240.254:37549 was elected leader, sending a full tablet report...
W20260810 11:37:05.556152 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:05.556488 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:05.556669 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:05.564354 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:05.564512 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:05.564688 6198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:05.564857 6198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:05.575165 6198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34351
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:37549
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:05.576548 6198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:05.577931 6198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:05.592386 6206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:05.592487 6209 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:05.592468 6205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:05.593950 6198 server_base.cc:1034] running on GCE node
I20260810 11:37:05.594854 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:05.596158 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:05.597487 6198 hybrid_clock.cc:648] HybridClock initialized: now 1786361825597359 us; error 97 us; skew 500 ppm
I20260810 11:37:05.597863 6198 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:05 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:05.601513 6198 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:05.603195 6198 webserver.cc:460] Webserver started at http://127.3.240.195:32797/ using document root <none> and password file <none>
I20260810 11:37:05.603905 6198 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:05.604038 6198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:05.604457 6198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:05.607653 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "729d8b497961433e88b50276cdd15898"
format_stamp: "Formatted at 2026-08-10 11:37:05 on dist-test-slave-drl0"
server_key: "139b3e4fded864f15fab39ca4968900a"
server_key_iv: "c5e9d04040789d664b4ee8f031b26a1a"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.608505 6198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "729d8b497961433e88b50276cdd15898"
format_stamp: "Formatted at 2026-08-10 11:37:05 on dist-test-slave-drl0"
server_key: "139b3e4fded864f15fab39ca4968900a"
server_key_iv: "c5e9d04040789d664b4ee8f031b26a1a"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.614524 6198 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 11:37:05.618680 6216 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:05.620738 6198 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:05.621066 6198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "729d8b497961433e88b50276cdd15898"
format_stamp: "Formatted at 2026-08-10 11:37:05 on dist-test-slave-drl0"
server_key: "139b3e4fded864f15fab39ca4968900a"
server_key_iv: "c5e9d04040789d664b4ee8f031b26a1a"
server_key_version: "encryptionkey@0"
I20260810 11:37:05.621374 6198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:05.652061 6198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:05.657572 6198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:05.657902 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:05.660800 6198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:05.660910 6198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:05.661095 6198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:05.661191 6198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:05 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:05.691100 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.661368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48949 (local address 127.3.240.254:37549)
0810 11:37:05.662104 (+ 736us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.662111 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.663216 (+ 1105us) server_negotiation.cc:407] Connection header received
0810 11:37:05.666429 (+ 3213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.666436 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.666542 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.666702 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:05.669000 (+ 2298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.670344 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.672812 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.673213 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.673317 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.679822 (+ 6505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:05.679833 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:05.679840 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:05.679889 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:05.684357 (+ 4468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.685992 (+ 1635us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.686000 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.686004 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.686165 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.686952 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.686960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.686966 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.687389 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:05.687574 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.690028 (+ 2454us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.690318 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":485,"threads_started":1}
I20260810 11:37:05.692572 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.662060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.195:48949)
0810 11:37:05.663005 (+ 945us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.663068 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.666049 (+ 2981us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.667043 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.667059 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.667896 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:05.668708 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.668727 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.670547 (+ 1820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.670552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.672580 (+ 2028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.672598 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.673507 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.673518 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.673913 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.674920 (+ 1007us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:05.675140 (+ 220us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:05.679459 (+ 4319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:05.684526 (+ 5067us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.684542 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.684563 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.685730 (+ 1167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.686369 (+ 639us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.686378 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.686385 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.686746 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.687577 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:05.687592 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:05.688234 (+ 642us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.688609 (+ 375us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.689431 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":366,"threads_started":1}
I20260810 11:37:05.712030 6198 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:39229
I20260810 11:37:05.712081 6388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:39229 every 8 connection(s)
I20260810 11:37:05.714075 6198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:05.716100 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 6198
I20260810 11:37:05.716331 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:05.717062 4035 external_mini_cluster.cc:1428] Setting key 39b11465f4f24edb758113e06342ba20
I20260810 11:37:05.717424 6389 heartbeater.cc:347] Connected to a master server at 127.3.240.254:37549
I20260810 11:37:05.718670 6389 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:05.719477 6389 heartbeater.cc:511] Master 127.3.240.254:37549 requested a full tablet report, sending...
I20260810 11:37:05.721503 5695 ts_manager.cc:194] Registered new tserver with Master: 729d8b497961433e88b50276cdd15898 (127.3.240.195:39229)
I20260810 11:37:05.722896 5695 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:48949
I20260810 11:37:05.723906 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.703914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38333 (local address 127.3.240.254:37549)
0810 11:37:05.704152 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.704161 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.707695 (+ 3534us) server_negotiation.cc:407] Connection header received
0810 11:37:05.707938 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.707942 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.708007 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.708129 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:05.709099 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.709962 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.712004 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.712627 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.712768 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.714874 (+ 2106us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:05.714885 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:05.714895 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:05.715000 (+ 105us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:05.720082 (+ 5082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.721001 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.721009 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.721012 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.721217 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.721710 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.721715 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.721718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.722036 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:05.722251 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.723089 (+ 838us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.723418 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":2,"server-negotiator.queue_time_us":56}
I20260810 11:37:05.723986 6369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.704060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.3.240.195:38333)
0810 11:37:05.707547 (+ 3487us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.707578 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.707762 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.708166 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.708170 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.708651 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:05.708945 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.708955 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.710123 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.710128 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.711819 (+ 1691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.711845 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.712876 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.712881 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.713007 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.713690 (+ 683us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:05.713708 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:05.714558 (+ 850us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:05.720314 (+ 5756us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.720320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.720324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.720843 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.721377 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:05.721383 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:05.721386 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:05.721531 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:05.722184 (+ 653us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:05.722190 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:05.722615 (+ 425us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.722882 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.723431 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3269,"thread_start_us":3092,"threads_started":1}
I20260810 11:37:05.734969 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 11:37:05 dist-test-slave-drl0 krb5kdc[5633](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 11:37:05 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:05.801906 6231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.782642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37610 (local address 127.3.240.254:37549)
0810 11:37:05.783080 (+ 438us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.783086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.783109 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:05.783235 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.783240 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.783319 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.783428 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:05.785290 (+ 1862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.786290 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.788000 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.788703 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.788853 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.794317 (+ 5464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:05.794330 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:05.794335 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:05.794389 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:05.798546 (+ 4157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.799618 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.799622 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.799625 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.799768 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:05.800168 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:05.800173 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:05.800175 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:05.800475 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:05.800770 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.801275 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.801557 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":206}
I20260810 11:37:05.807720 5695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:05.815023 5695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:05.853518 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.830086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57092 (local address 127.3.240.194:40359)
0810 11:37:05.843243 (+ 13157us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.843258 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.844137 (+ 879us) server_negotiation.cc:407] Connection header received
0810 11:37:05.844454 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.844465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.844746 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.844959 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:05.846005 (+ 1046us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.847129 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.849940 (+ 2811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.851391 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.851534 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.851744 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.853088 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.853311 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12905,"thread_start_us":12966,"threads_started":1}
I20260810 11:37:05.853747 6412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.829927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:40359 (local address 127.0.0.1:57092)
0810 11:37:05.843949 (+ 14022us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.843993 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.844208 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.845272 (+ 1064us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.845279 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.845316 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:05.845813 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.845829 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.847294 (+ 1465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.847300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.849744 (+ 2444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.849772 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.851607 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.851856 (+ 249us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.852096 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.852153 (+ 57us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.852326 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.852530 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13733,"thread_start_us":13582,"threads_started":1}
I20260810 11:37:05.854815 6415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.832815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:39229 (local address 127.0.0.1:46654)
0810 11:37:05.845587 (+ 12772us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.845607 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.845832 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.846866 (+ 1034us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.846874 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.846905 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:05.847294 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.847305 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.849335 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.849341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.851136 (+ 1795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.851150 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.853891 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.853898 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.854087 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.854138 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.854229 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.854359 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12641,"mutex_wait_us":49,"thread_start_us":11949,"threads_started":1}
I20260810 11:37:05.855615 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.833645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46654 (local address 127.3.240.195:39229)
0810 11:37:05.844943 (+ 11298us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.844953 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.845711 (+ 758us) server_negotiation.cc:407] Connection header received
0810 11:37:05.846058 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.846066 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.846284 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.846473 (+ 189us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:05.847552 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.849179 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.851319 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.852836 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.853004 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.853237 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.855103 (+ 1866us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.855269 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11044,"thread_start_us":10819,"threads_started":1}
I20260810 11:37:05.860349 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.830501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53662 (local address 127.3.240.193:44885)
0810 11:37:05.842969 (+ 12468us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:05.842985 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:05.845337 (+ 2352us) server_negotiation.cc:407] Connection header received
0810 11:37:05.845497 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:05.845507 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:05.845755 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:05.845963 (+ 208us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:05.846972 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.848479 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.851101 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.852567 (+ 1466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.852697 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.852889 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:05.859855 (+ 6966us) server_negotiation.cc:299] Negotiation successful
0810 11:37:05.860069 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12121,"thread_start_us":11860,"threads_started":1}
I20260810 11:37:05.861070 6417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:05.830085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44885 (local address 127.0.0.1:53662)
0810 11:37:05.844862 (+ 14777us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:05.844892 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:05.845038 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:05.846244 (+ 1206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:05.846250 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:05.846275 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:05.846749 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.846765 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.848677 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.848683 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.850892 (+ 2209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:05.850908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:05.853838 (+ 2930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:05.853853 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:05.854102 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:05.854155 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 11:37:05.854258 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 11:37:05.860666 (+ 6408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14623,"mutex_wait_us":55,"thread_start_us":15157,"threads_started":1}
I20260810 11:37:05.860730 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 (DEFAULT_TABLE table=test-table [id=1e41fe9800d4431abdcf75dc685dd718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:05.861055 6088 tablet_service.cc:1467] Processing CreateTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 (DEFAULT_TABLE table=test-table [id=1e41fe9800d4431abdcf75dc685dd718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:05.863137 6088 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b1ff418c744bc8ce00c03c93d3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:05.863147 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b1ff418c744bc8ce00c03c93d3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:05.866976 5894 tablet_service.cc:1467] Processing CreateTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 (DEFAULT_TABLE table=test-table [id=1e41fe9800d4431abdcf75dc685dd718]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:05.868413 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 885b1ff418c744bc8ce00c03c93d3bd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:05.886919 6423 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Bootstrap starting.
I20260810 11:37:05.887372 6420 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Bootstrap starting.
I20260810 11:37:05.888419 6424 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Bootstrap starting.
I20260810 11:37:05.893029 6424 tablet_bootstrap.cc:654] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:05.894596 6423 tablet_bootstrap.cc:654] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:05.894846 6424 log.cc:824] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:05.896075 6420 tablet_bootstrap.cc:654] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:05.897607 6423 log.cc:824] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:05.899150 6420 log.cc:824] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:05.906987 6423 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: No bootstrap required, opened a new log
I20260810 11:37:05.907341 6423 ts_tablet_manager.cc:1397] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Time spent bootstrapping tablet: real 0.021s user 0.008s sys 0.009s
I20260810 11:37:05.908839 6424 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: No bootstrap required, opened a new log
I20260810 11:37:05.909329 6424 ts_tablet_manager.cc:1397] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.007s
I20260810 11:37:05.909857 6420 tablet_bootstrap.cc:492] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: No bootstrap required, opened a new log
I20260810 11:37:05.910382 6420 ts_tablet_manager.cc:1397] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.003s
I20260810 11:37:05.921072 6423 raft_consensus.cc:348] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.921955 6423 raft_consensus.cc:374] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:05.922099 6423 raft_consensus.cc:729] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6daa2b29536140e8b7ede072e722b862, State: Initialized, Role: FOLLOWER
I20260810 11:37:05.922922 6423 consensus_queue.cc:260] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.924345 6194 heartbeater.cc:503] Master 127.3.240.254:37549 was elected leader, sending a full tablet report...
I20260810 11:37:05.925227 6423 ts_tablet_manager.cc:1428] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
I20260810 11:37:05.927423 6420 raft_consensus.cc:348] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.928166 6420 raft_consensus.cc:374] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:05.928323 6420 raft_consensus.cc:729] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d8b497961433e88b50276cdd15898, State: Initialized, Role: FOLLOWER
I20260810 11:37:05.929131 6420 consensus_queue.cc:260] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.929561 6424 raft_consensus.cc:348] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.930280 6424 raft_consensus.cc:374] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:05.930537 6424 raft_consensus.cc:729] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39022873c814a00bed1404afeda2505, State: Initialized, Role: FOLLOWER
I20260810 11:37:05.931346 6389 heartbeater.cc:503] Master 127.3.240.254:37549 was elected leader, sending a full tablet report...
I20260810 11:37:05.931643 6424 consensus_queue.cc:260] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:05.932437 6420 ts_tablet_manager.cc:1428] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Time spent starting tablet: real 0.022s user 0.015s sys 0.008s
I20260810 11:37:05.934909 6424 ts_tablet_manager.cc:1428] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Time spent starting tablet: real 0.025s user 0.018s sys 0.007s
W20260810 11:37:05.966540 6392 tablet.cc:2369] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:06.027815 5994 tablet.cc:2369] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:06.105095 6431 raft_consensus.cc:482] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:06.105576 6431 raft_consensus.cc:504] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
W20260810 11:37:06.107780 6195 tablet.cc:2369] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:06.109654 6431 leader_election.cc:288] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359), a39022873c814a00bed1404afeda2505 (127.3.240.193:44885)
I20260810 11:37:06.118178 6413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.109767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:60697 (local address 127.3.240.194:40359)
0810 11:37:06.110124 (+ 357us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.110142 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.110186 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:37:06.110319 (+ 133us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.110328 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.110465 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.110619 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:06.111147 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.112326 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.114777 (+ 2451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.116913 (+ 2136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.117126 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.117252 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.117723 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.117854 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 11:37:06.118221 6232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.108616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:40359 (local address 127.3.240.195:60697)
0810 11:37:06.109327 (+ 711us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:06.109351 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:06.109548 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:06.110650 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:06.110655 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:06.110690 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:06.110968 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.110982 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.112500 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.112505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.114517 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.114532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.117094 (+ 2562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.117104 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.117442 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.117481 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 11:37:06.117616 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 11:37:06.117785 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":438}
I20260810 11:37:06.119720 6433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.109139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44885 (local address 127.3.240.195:52555)
0810 11:37:06.111003 (+ 1864us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:06.111024 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:06.111268 (+ 244us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:06.111868 (+ 600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:06.111873 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:06.111902 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:06.112271 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.112280 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.113579 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.113587 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.116826 (+ 3239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.116844 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.118662 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.118669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.118838 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.118868 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 11:37:06.119136 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 11:37:06.119347 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1612,"thread_start_us":1568,"threads_started":1}
I20260810 11:37:06.119726 6416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.109063 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:52555 (local address 127.3.240.193:44885)
0810 11:37:06.109308 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.109314 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.111119 (+ 1805us) server_negotiation.cc:407] Connection header received
0810 11:37:06.111432 (+ 313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.111438 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.111525 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.111651 (+ 126us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:06.112446 (+ 795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.113418 (+ 972us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.117025 (+ 3607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.118280 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.118440 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.118510 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.119172 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.119362 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:37:06.119949 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b1ff418c744bc8ce00c03c93d3bd0" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862" is_pre_election: true
I20260810 11:37:06.121133 6121 raft_consensus.cc:2454] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.121439 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b1ff418c744bc8ce00c03c93d3bd0" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505" is_pre_election: true
I20260810 11:37:06.122356 5928 raft_consensus.cc:2454] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.122591 6220 leader_election.cc:302] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6daa2b29536140e8b7ede072e722b862, 729d8b497961433e88b50276cdd15898; no voters:
I20260810 11:37:06.123348 6431 raft_consensus.cc:2781] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:06.123631 6431 raft_consensus.cc:482] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:06.123842 6431 raft_consensus.cc:3037] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.127565 6431 raft_consensus.cc:504] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.128409 6431 leader_election.cc:288] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Requested vote from peers 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359), a39022873c814a00bed1404afeda2505 (127.3.240.193:44885)
I20260810 11:37:06.129050 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b1ff418c744bc8ce00c03c93d3bd0" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505"
I20260810 11:37:06.129108 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "885b1ff418c744bc8ce00c03c93d3bd0" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862"
I20260810 11:37:06.129359 6121 raft_consensus.cc:3037] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.129390 5928 raft_consensus.cc:3037] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.133116 6121 raft_consensus.cc:2454] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.133780 6220 leader_election.cc:302] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6daa2b29536140e8b7ede072e722b862, 729d8b497961433e88b50276cdd15898; no voters:
I20260810 11:37:06.134230 6431 raft_consensus.cc:2781] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:06.135025 5928 raft_consensus.cc:2454] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.135404 6431 raft_consensus.cc:686] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 1 LEADER]: Becoming Leader. State: Replica: 729d8b497961433e88b50276cdd15898, State: Running, Role: LEADER
I20260810 11:37:06.136199 6431 consensus_queue.cc:237] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.142720 5697 catalog_manager.cc:5495] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 reported cstate change: term changed from 0 to 1, leader changed from <none> to 729d8b497961433e88b50276cdd15898 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } health_report { overall_health: HEALTHY } } }
I20260810 11:37:06.191325 6418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.183089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46662 (local address 127.3.240.195:39229)
0810 11:37:06.183356 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.183362 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.183388 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:37:06.183660 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.183665 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.183781 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.183918 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:06.184500 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.185761 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.187831 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.188784 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.188963 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.189689 (+ 726us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:06.190308 (+ 619us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:06.190429 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.190680 (+ 251us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.190872 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 11:37:06.202551 5928 raft_consensus.cc:1261] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.202764 6121 raft_consensus.cc:1261] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.205884 6435 consensus_queue.cc:1035] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260810 11:37:06.206624 6431 consensus_queue.cc:1035] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:06.223237 6438 mvcc.cc:205] Tried to move back new op lower bound from 7316938040113385472 to 7316938039858450432. Current Snapshot: MvccSnapshot[applied={T|T < 7316938040113385472}]
I20260810 11:37:06.256070 5695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:06.256559 5695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:06.262359 5695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Sending DeleteTablet for 3 replicas of tablet 885b1ff418c744bc8ce00c03c93d3bd0
I20260810 11:37:06.264103 6088 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57092
I20260810 11:37:06.264326 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53662
I20260810 11:37:06.265278 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet 885b1ff418c744bc8ce00c03c93d3bd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:06 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46654
I20260810 11:37:06.267597 6452 tablet_replica.cc:330] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: stopping tablet replica
I20260810 11:37:06.267899 6453 tablet_replica.cc:330] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: stopping tablet replica
I20260810 11:37:06.268333 6452 raft_consensus.cc:2229] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:06.268461 6453 raft_consensus.cc:2229] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:06.268648 5695 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 11:37:06.269245 6452 raft_consensus.cc:2258] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 11:37:06.269510 5695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:06.269984 6453 raft_consensus.cc:2258] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:06.272225 6452 ts_tablet_manager.cc:1905] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:06.274950 6453 ts_tablet_manager.cc:1905] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:06.282346 6452 ts_tablet_manager.cc:1918] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:06.282620 6452 log.cc:1196] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/885b1ff418c744bc8ce00c03c93d3bd0
I20260810 11:37:06.283368 6452 ts_tablet_manager.cc:1939] T 885b1ff418c744bc8ce00c03c93d3bd0 P a39022873c814a00bed1404afeda2505: Deleting consensus metadata
I20260810 11:37:06.285467 6298 tablet_service.cc:1467] Processing CreateTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 (DEFAULT_TABLE table=test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:06.286350 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1f2a00e14d404b9729855bfb6ed805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.286852 5677 catalog_manager.cc:4841] TS a39022873c814a00bed1404afeda2505 (127.3.240.193:44885): tablet 885b1ff418c744bc8ce00c03c93d3bd0 (table test-table [id=1e41fe9800d4431abdcf75dc685dd718]) successfully deleted
I20260810 11:37:06.288010 5894 tablet_service.cc:1467] Processing CreateTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 (DEFAULT_TABLE table=test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:06.288609 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1f2a00e14d404b9729855bfb6ed805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.301156 6420 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Bootstrap starting.
I20260810 11:37:06.304325 6420 tablet_bootstrap.cc:654] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.306068 6424 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Bootstrap starting.
I20260810 11:37:06.305950 6087 tablet_service.cc:1467] Processing CreateTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 (DEFAULT_TABLE table=test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:06.307264 6087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ea1f2a00e14d404b9729855bfb6ed805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.308938 6424 tablet_bootstrap.cc:654] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.309201 6453 ts_tablet_manager.cc:1918] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:06.309610 6453 log.cc:1196] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/885b1ff418c744bc8ce00c03c93d3bd0
I20260810 11:37:06.310878 6453 ts_tablet_manager.cc:1939] T 885b1ff418c744bc8ce00c03c93d3bd0 P 729d8b497961433e88b50276cdd15898: Deleting consensus metadata
I20260810 11:37:06.319973 6451 tablet_replica.cc:330] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: stopping tablet replica
I20260810 11:37:06.320968 6451 raft_consensus.cc:2229] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:06.322340 5678 catalog_manager.cc:4841] TS 729d8b497961433e88b50276cdd15898 (127.3.240.195:39229): tablet 885b1ff418c744bc8ce00c03c93d3bd0 (table test-table [id=1e41fe9800d4431abdcf75dc685dd718]) successfully deleted
I20260810 11:37:06.322960 6420 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: No bootstrap required, opened a new log
I20260810 11:37:06.323251 6420 ts_tablet_manager.cc:1397] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Time spent bootstrapping tablet: real 0.022s user 0.009s sys 0.001s
I20260810 11:37:06.323325 6451 raft_consensus.cc:2258] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:06.324604 6420 raft_consensus.cc:348] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.325013 6420 raft_consensus.cc:374] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.325254 6420 raft_consensus.cc:729] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d8b497961433e88b50276cdd15898, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.325697 6420 consensus_queue.cc:260] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.326169 6423 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Bootstrap starting.
I20260810 11:37:06.327879 6420 ts_tablet_manager.cc:1428] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:06.328605 6423 tablet_bootstrap.cc:654] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.329479 6451 ts_tablet_manager.cc:1905] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:06.335578 6424 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: No bootstrap required, opened a new log
I20260810 11:37:06.335981 6424 ts_tablet_manager.cc:1397] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Time spent bootstrapping tablet: real 0.030s user 0.004s sys 0.012s
I20260810 11:37:06.337373 6424 raft_consensus.cc:348] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.337795 6424 raft_consensus.cc:374] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.337975 6424 raft_consensus.cc:729] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39022873c814a00bed1404afeda2505, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.338420 6424 consensus_queue.cc:260] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.343641 6451 ts_tablet_manager.cc:1918] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:06.343931 6451 log.cc:1196] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/885b1ff418c744bc8ce00c03c93d3bd0
I20260810 11:37:06.344278 6424 ts_tablet_manager.cc:1428] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260810 11:37:06.345175 6451 ts_tablet_manager.cc:1939] T 885b1ff418c744bc8ce00c03c93d3bd0 P 6daa2b29536140e8b7ede072e722b862: Deleting consensus metadata
I20260810 11:37:06.347939 5681 catalog_manager.cc:4841] TS 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359): tablet 885b1ff418c744bc8ce00c03c93d3bd0 (table test-table [id=1e41fe9800d4431abdcf75dc685dd718]) successfully deleted
I20260810 11:37:06.348978 6423 tablet_bootstrap.cc:492] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: No bootstrap required, opened a new log
I20260810 11:37:06.349228 6423 ts_tablet_manager.cc:1397] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Time spent bootstrapping tablet: real 0.023s user 0.009s sys 0.012s
I20260810 11:37:06.350700 6423 raft_consensus.cc:348] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.350962 6423 raft_consensus.cc:374] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.351150 6423 raft_consensus.cc:729] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6daa2b29536140e8b7ede072e722b862, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.351477 6423 consensus_queue.cc:260] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.352190 6423 ts_tablet_manager.cc:1428] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 11:37:06.433434 6448 raft_consensus.cc:482] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:06.433883 6448 raft_consensus.cc:504] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.435284 6448 leader_election.cc:288] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39022873c814a00bed1404afeda2505 (127.3.240.193:44885), 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359)
I20260810 11:37:06.436055 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1f2a00e14d404b9729855bfb6ed805" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505" is_pre_election: true
I20260810 11:37:06.435999 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1f2a00e14d404b9729855bfb6ed805" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862" is_pre_election: true
I20260810 11:37:06.436352 6121 raft_consensus.cc:2454] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.436399 5928 raft_consensus.cc:2454] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.436983 6220 leader_election.cc:302] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6daa2b29536140e8b7ede072e722b862, 729d8b497961433e88b50276cdd15898; no voters:
I20260810 11:37:06.437453 6448 raft_consensus.cc:2781] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:06.437633 6448 raft_consensus.cc:482] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:06.438073 6448 raft_consensus.cc:3037] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.443951 6448 raft_consensus.cc:504] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.444950 6448 leader_election.cc:288] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Requested vote from peers a39022873c814a00bed1404afeda2505 (127.3.240.193:44885), 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359)
I20260810 11:37:06.445492 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1f2a00e14d404b9729855bfb6ed805" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862"
I20260810 11:37:06.445719 6121 raft_consensus.cc:3037] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.445657 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ea1f2a00e14d404b9729855bfb6ed805" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505"
I20260810 11:37:06.445962 5928 raft_consensus.cc:3037] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.449421 6121 raft_consensus.cc:2454] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.450253 6220 leader_election.cc:302] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6daa2b29536140e8b7ede072e722b862, 729d8b497961433e88b50276cdd15898; no voters:
I20260810 11:37:06.450747 5928 raft_consensus.cc:2454] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.450858 6448 raft_consensus.cc:2781] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:06.451148 6448 raft_consensus.cc:686] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 1 LEADER]: Becoming Leader. State: Replica: 729d8b497961433e88b50276cdd15898, State: Running, Role: LEADER
I20260810 11:37:06.451573 6448 consensus_queue.cc:237] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.455744 5695 catalog_manager.cc:5495] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 reported cstate change: term changed from 0 to 1, leader changed from <none> to 729d8b497961433e88b50276cdd15898 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } health_report { overall_health: HEALTHY } } }
Aug 10 11:37:06 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361823, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:06.557338 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.528474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37549 (local address 127.0.0.1:37618)
0810 11:37:06.531221 (+ 2747us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:06.531257 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:06.531505 (+ 248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:06.532083 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:06.532093 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:06.533315 (+ 1222us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:06.533781 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.533798 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.535984 (+ 2186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.535991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.539050 (+ 3059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.539097 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.541156 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.541163 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.541418 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.542462 (+ 1044us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:06.542496 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:06.547104 (+ 4608us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:06.553647 (+ 6543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:06.553673 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:06.553683 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:06.554609 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:06.555344 (+ 735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:06.555348 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:06.555352 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:06.555521 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:06.556351 (+ 830us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:06.556359 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:06.556557 (+ 198us) client_negotiation.cc:769] Sending connection context
0810 11:37:06.556730 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 11:37:06.556980 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2535,"thread_start_us":2291,"threads_started":1}
I20260810 11:37:06.557811 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.528364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37618 (local address 127.3.240.254:37549)
0810 11:37:06.529369 (+ 1005us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.529375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.531398 (+ 2023us) server_negotiation.cc:407] Connection header received
0810 11:37:06.531773 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.531785 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.531901 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.532018 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:06.534013 (+ 1995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.535813 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.539410 (+ 3597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.540812 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.541001 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.547510 (+ 6509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:06.547520 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:06.547526 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:06.547584 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:06.553285 (+ 5701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:06.554843 (+ 1558us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:06.554849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:06.554854 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:06.555173 (+ 319us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:06.555705 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:06.555711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:06.555714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:06.556176 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:06.556326 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.556908 (+ 582us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.557186 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":773,"thread_start_us":633,"threads_started":1}
I20260810 11:37:06.565951 5695 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:37618:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 11:37:06.568110 5695 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:06.578192 5894 tablet_service.cc:1467] Processing CreateTablet for tablet 275d8db18f834ae8b2d606f5f0272fc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15fe6520ecaf4c72a5f654f2636f049f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:06.578998 5894 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d8db18f834ae8b2d606f5f0272fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.578492 6087 tablet_service.cc:1467] Processing CreateTablet for tablet 275d8db18f834ae8b2d606f5f0272fc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15fe6520ecaf4c72a5f654f2636f049f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:06.579659 6087 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d8db18f834ae8b2d606f5f0272fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.584960 6298 tablet_service.cc:1467] Processing CreateTablet for tablet 275d8db18f834ae8b2d606f5f0272fc3 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=15fe6520ecaf4c72a5f654f2636f049f]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:06.585922 6298 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 275d8db18f834ae8b2d606f5f0272fc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:06.589721 6424 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: Bootstrap starting.
I20260810 11:37:06.590592 6423 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: Bootstrap starting.
I20260810 11:37:06.592852 6424 tablet_bootstrap.cc:654] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.593961 6423 tablet_bootstrap.cc:654] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.600453 6424 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: No bootstrap required, opened a new log
I20260810 11:37:06.600765 6424 ts_tablet_manager.cc:1397] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.001s
I20260810 11:37:06.601516 6423 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: No bootstrap required, opened a new log
I20260810 11:37:06.601573 6420 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: Bootstrap starting.
I20260810 11:37:06.601774 6423 ts_tablet_manager.cc:1397] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.004s
I20260810 11:37:06.601974 6424 raft_consensus.cc:348] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.602339 6424 raft_consensus.cc:374] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.602429 6424 raft_consensus.cc:729] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a39022873c814a00bed1404afeda2505, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.602887 6423 raft_consensus.cc:348] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.603036 6424 consensus_queue.cc:260] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.603250 6423 raft_consensus.cc:374] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.603410 6423 raft_consensus.cc:729] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6daa2b29536140e8b7ede072e722b862, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.603690 6432 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.604244 6432 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.604085 6423 consensus_queue.cc:260] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.604538 6420 tablet_bootstrap.cc:654] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:06.604816 6429 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.605612 6429 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.607209 6423 ts_tablet_manager.cc:1428] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: Time spent starting tablet: real 0.005s user 0.003s sys 0.000s
I20260810 11:37:06.609196 6424 ts_tablet_manager.cc:1428] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: Time spent starting tablet: real 0.008s user 0.000s sys 0.003s
I20260810 11:37:06.609346 6420 tablet_bootstrap.cc:492] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: No bootstrap required, opened a new log
I20260810 11:37:06.609683 6420 ts_tablet_manager.cc:1397] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.001s
I20260810 11:37:06.611958 6420 raft_consensus.cc:348] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.612350 6420 raft_consensus.cc:374] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:06.612534 6420 raft_consensus.cc:729] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 729d8b497961433e88b50276cdd15898, State: Initialized, Role: FOLLOWER
I20260810 11:37:06.612891 6420 consensus_queue.cc:260] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.613444 6448 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.613973 6420 ts_tablet_manager.cc:1428] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260810 11:37:06.614090 6448 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.671792 6448 raft_consensus.cc:482] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:06.672178 6448 raft_consensus.cc:504] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.673475 6448 leader_election.cc:288] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a39022873c814a00bed1404afeda2505 (127.3.240.193:44885), 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359)
I20260810 11:37:06.673858 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d8db18f834ae8b2d606f5f0272fc3" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505" is_pre_election: true
I20260810 11:37:06.674259 5928 raft_consensus.cc:2454] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.674662 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d8db18f834ae8b2d606f5f0272fc3" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862" is_pre_election: true
I20260810 11:37:06.674804 6218 leader_election.cc:302] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729d8b497961433e88b50276cdd15898, a39022873c814a00bed1404afeda2505; no voters:
I20260810 11:37:06.675024 6121 raft_consensus.cc:2454] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 0.
I20260810 11:37:06.675181 6448 raft_consensus.cc:2781] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:06.675354 6448 raft_consensus.cc:482] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:06.675673 6448 raft_consensus.cc:3037] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.686240 6448 raft_consensus.cc:504] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.687933 6121 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d8db18f834ae8b2d606f5f0272fc3" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6daa2b29536140e8b7ede072e722b862"
I20260810 11:37:06.687922 5928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "275d8db18f834ae8b2d606f5f0272fc3" candidate_uuid: "729d8b497961433e88b50276cdd15898" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a39022873c814a00bed1404afeda2505"
I20260810 11:37:06.688241 6121 raft_consensus.cc:3037] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.688285 5928 raft_consensus.cc:3037] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:06.692063 5928 raft_consensus.cc:2454] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.692387 6448 leader_election.cc:288] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Requested vote from peers a39022873c814a00bed1404afeda2505 (127.3.240.193:44885), 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359)
I20260810 11:37:06.692597 6218 leader_election.cc:302] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 729d8b497961433e88b50276cdd15898, a39022873c814a00bed1404afeda2505; no voters:
I20260810 11:37:06.692689 6121 raft_consensus.cc:2454] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 729d8b497961433e88b50276cdd15898 in term 1.
I20260810 11:37:06.693071 6448 raft_consensus.cc:2781] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:06.693692 6448 raft_consensus.cc:686] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [term 1 LEADER]: Becoming Leader. State: Replica: 729d8b497961433e88b50276cdd15898, State: Running, Role: LEADER
I20260810 11:37:06.694052 6448 consensus_queue.cc:237] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } }
I20260810 11:37:06.697064 6431 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: TxnStatusTablet state changed. Reason: New leader 729d8b497961433e88b50276cdd15898. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.697392 6431 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:06.699290 5697 catalog_manager.cc:5495] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 reported cstate change: term changed from 0 to 1, leader changed from <none> to 729d8b497961433e88b50276cdd15898 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } health_report { overall_health: HEALTHY } } }
I20260810 11:37:06.707934 6477 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 11:37:06.708302 6477 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 11:37:06.766870 6468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.751139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:39229 (local address 127.0.0.1:46674)
0810 11:37:06.751555 (+ 416us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:06.751577 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:06.751755 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:06.759038 (+ 7283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:06.759045 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:06.759062 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:06.759437 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.759455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.762040 (+ 2585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.762057 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.764012 (+ 1955us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.764041 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.764930 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.764940 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.765220 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.765287 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:06.765800 (+ 513us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:06.765825 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:37:06.765924 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 11:37:06.766127 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":225}
I20260810 11:37:06.766816 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.745149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46674 (local address 127.3.240.195:39229)
0810 11:37:06.758424 (+ 13275us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.758430 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.758458 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:37:06.758553 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.758561 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.758650 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.759606 (+ 956us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:06.760059 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.761829 (+ 1770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.764235 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.764710 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.764861 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.765561 (+ 700us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:06.765676 (+ 115us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:06.765796 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.765964 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.766045 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13142,"thread_start_us":13606,"threads_started":1}
I20260810 11:37:06.773205 6121 raft_consensus.cc:1261] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.773183 5928 raft_consensus.cc:1261] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.774006 6431 consensus_queue.cc:1035] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:06.774757 6448 consensus_queue.cc:1035] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:06.795928 6429 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: TxnStatusTablet state changed. Reason: New leader 729d8b497961433e88b50276cdd15898. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.796399 6429 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.797269 6432 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: TxnStatusTablet state changed. Reason: New leader 729d8b497961433e88b50276cdd15898. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.797719 6432 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.810045 6481 mvcc.cc:205] Tried to move back new op lower bound from 7316938042454634496 to 7316938042142060544. Current Snapshot: MvccSnapshot[applied={T|T < 7316938042454634496}]
I20260810 11:37:06.813898 6432 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.814304 6432 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P a39022873c814a00bed1404afeda2505: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.815834 6431 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.820425 6431 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:06.820796 6431 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.821448 6431 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 729d8b497961433e88b50276cdd15898: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:06.826913 6429 tablet_replica.cc:439] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "729d8b497961433e88b50276cdd15898" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 } } peers { permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 } } peers { permanent_uuid: "729d8b497961433e88b50276cdd15898" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 39229 } } }
I20260810 11:37:06.827453 6429 tablet_replica.cc:442] T 275d8db18f834ae8b2d606f5f0272fc3 P 6daa2b29536140e8b7ede072e722b862: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:06.866324 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.853893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55547 (local address 127.3.240.195:39229)
0810 11:37:06.854095 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:06.854104 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:06.856284 (+ 2180us) server_negotiation.cc:407] Connection header received
0810 11:37:06.856641 (+ 357us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:06.856648 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:06.856740 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:06.856803 (+ 63us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:06.858332 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.860193 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.863321 (+ 3128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.864331 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.864451 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.865287 (+ 836us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:06.865463 (+ 176us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:06.865575 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:06.865771 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 11:37:06.865904 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":60}
I20260810 11:37:06.866727 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:06.853788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:39229 (local address 127.3.240.195:55547)
0810 11:37:06.856167 (+ 2379us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:06.856192 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:06.856477 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:06.857029 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:06.857036 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:06.857051 (+ 15us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:06.858007 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.858025 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.860350 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.860356 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.863129 (+ 2773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:06.863158 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:06.864697 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:06.864704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:06.864929 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:06.865023 (+ 94us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:06.865599 (+ 576us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:06.865613 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 11:37:06.865724 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 11:37:06.865911 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2172,"thread_start_us":1905,"threads_started":1}
I20260810 11:37:06.885473 6118 raft_consensus.cc:1261] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.886209 5928 raft_consensus.cc:1261] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Refusing update from remote peer 729d8b497961433e88b50276cdd15898: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:06.886521 6493 consensus_queue.cc:1035] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6daa2b29536140e8b7ede072e722b862" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40359 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260810 11:37:06.887892 6448 consensus_queue.cc:1035] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a39022873c814a00bed1404afeda2505" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44885 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:06.918361 6482 mvcc.cc:205] Tried to move back new op lower bound from 7316938042913705984 to 7316938041148272640. Current Snapshot: MvccSnapshot[applied={T|T < 7316938042913705984}]
I20260810 11:37:07.026782 5695 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:07.027199 5695 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37610:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:07.034080 5695 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0d7117160b044926acab7671647c1c4d: Sending DeleteTablet for 3 replicas of tablet ea1f2a00e14d404b9729855bfb6ed805
I20260810 11:37:07.035497 5894 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:07 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53662
I20260810 11:37:07.035543 6087 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:07 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57092
I20260810 11:37:07.037307 6298 tablet_service.cc:1514] Processing DeleteTablet for tablet ea1f2a00e14d404b9729855bfb6ed805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:07 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46654
I20260810 11:37:07.052074 6506 tablet_replica.cc:330] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: stopping tablet replica
I20260810 11:37:07.052413 6506 raft_consensus.cc:2229] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:07.055079 6503 tablet_replica.cc:330] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: stopping tablet replica
I20260810 11:37:07.055480 6503 raft_consensus.cc:2229] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:07.053083 6506 raft_consensus.cc:2258] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:07.055961 6503 raft_consensus.cc:2258] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:07.057503 6504 tablet_replica.cc:330] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: stopping tablet replica
I20260810 11:37:07.057785 6506 ts_tablet_manager.cc:1905] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:07.057811 6504 raft_consensus.cc:2229] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:07.058425 6504 raft_consensus.cc:2258] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:07.060129 6503 ts_tablet_manager.cc:1905] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:07.063421 6504 ts_tablet_manager.cc:1905] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:07.078646 6506 ts_tablet_manager.cc:1918] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:07.079238 6506 log.cc:1196] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/ea1f2a00e14d404b9729855bfb6ed805
I20260810 11:37:07.080382 6506 ts_tablet_manager.cc:1939] T ea1f2a00e14d404b9729855bfb6ed805 P 729d8b497961433e88b50276cdd15898: Deleting consensus metadata
I20260810 11:37:07.084218 6503 ts_tablet_manager.cc:1918] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:07.084187 6504 ts_tablet_manager.cc:1918] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:07.084442 6504 log.cc:1196] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/ea1f2a00e14d404b9729855bfb6ed805
I20260810 11:37:07.084477 6503 log.cc:1196] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/ea1f2a00e14d404b9729855bfb6ed805
I20260810 11:37:07.085438 6504 ts_tablet_manager.cc:1939] T ea1f2a00e14d404b9729855bfb6ed805 P 6daa2b29536140e8b7ede072e722b862: Deleting consensus metadata
I20260810 11:37:07.085465 6503 ts_tablet_manager.cc:1939] T ea1f2a00e14d404b9729855bfb6ed805 P a39022873c814a00bed1404afeda2505: Deleting consensus metadata
I20260810 11:37:07.087288 5677 catalog_manager.cc:4841] TS a39022873c814a00bed1404afeda2505 (127.3.240.193:44885): tablet ea1f2a00e14d404b9729855bfb6ed805 (table test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]) successfully deleted
I20260810 11:37:07.087641 5681 catalog_manager.cc:4841] TS 6daa2b29536140e8b7ede072e722b862 (127.3.240.194:40359): tablet ea1f2a00e14d404b9729855bfb6ed805 (table test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]) successfully deleted
I20260810 11:37:07.089058 5678 catalog_manager.cc:4841] TS 729d8b497961433e88b50276cdd15898 (127.3.240.195:39229): tablet ea1f2a00e14d404b9729855bfb6ed805 (table test-table [id=03f277f3eb9e4eb0a1d9fc43bb4a8bfa]) successfully deleted
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:07.105448 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:07.049572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53672 (local address 127.3.240.193:44885)
0810 11:37:07.069533 (+ 19961us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:07.069542 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:07.069585 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:37:07.069656 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:07.069662 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:07.069753 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:07.069887 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:07.073770 (+ 3883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.076533 (+ 2763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.080256 (+ 3723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.081813 (+ 1557us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.083229 (+ 1416us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:07.092090 (+ 8861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:07.092114 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:07.092143 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:07.092201 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:07.098965 (+ 6764us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.100814 (+ 1849us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.100828 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.100832 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.101069 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.101759 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.101764 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.101767 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.102324 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:07.102692 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:07.104303 (+ 1611us) server_negotiation.cc:299] Negotiation successful
0810 11:37:07.104727 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19748,"thread_start_us":18217,"threads_started":1}
W20260810 11:37:07.106678 5987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53672
I20260810 11:37:07.160178 6512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:07.119741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.3.240.254:37549)
0810 11:37:07.130395 (+ 10654us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:07.130404 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:07.130431 (+ 27us) server_negotiation.cc:407] Connection header received
0810 11:37:07.130512 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:07.130521 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:07.130656 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:07.130799 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:07.134798 (+ 3999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.137130 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.140693 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.141913 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.142058 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:07.148142 (+ 6084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:07.148153 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:07.148161 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:07.148324 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:07.154422 (+ 6098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.155569 (+ 1147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.155577 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.155582 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.155806 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.156350 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.156358 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.156364 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.156831 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:07.157185 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:07.159159 (+ 1974us) server_negotiation.cc:299] Negotiation successful
0810 11:37:07.159553 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10492,"thread_start_us":8854,"threads_started":1}
W20260810 11:37:07.161582 5695 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37624
I20260810 11:37:07.165071 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5800
I20260810 11:37:07.192406 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:07.166606 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53688 (local address 127.3.240.193:44885)
0810 11:37:07.166833 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:07.166841 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:07.167130 (+ 289us) server_negotiation.cc:407] Connection header received
0810 11:37:07.167444 (+ 314us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:07.167450 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:07.167570 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:07.167650 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:07.169772 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.171114 (+ 1342us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.173855 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.174773 (+ 918us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.174967 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:07.180833 (+ 5866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:07.180842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:07.180851 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:07.180908 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:07.186304 (+ 5396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.188494 (+ 2190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.188513 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.188520 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.188762 (+ 242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.189451 (+ 689us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.189458 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.189463 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.189931 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:07.190372 (+ 441us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:07.190828 (+ 456us) server_negotiation.cc:299] Negotiation successful
0810 11:37:07.191163 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 11:37:07.193286 5987 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53688
W20260810 11:37:07.194183 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5800: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:07.194680 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5800
I20260810 11:37:07.227526 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6001
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:07.245067 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:07.228427 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.3.240.194:40359)
0810 11:37:07.229307 (+ 880us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:07.229312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:07.229327 (+ 15us) server_negotiation.cc:407] Connection header received
0810 11:37:07.229388 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:07.229392 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:07.229450 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:07.229581 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:07.231384 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.232120 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.233423 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.233878 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.234015 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:07.238686 (+ 4671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:07.238700 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:07.238721 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:07.238760 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:07.242129 (+ 3369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.242842 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.242851 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.242855 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.243075 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.243467 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.243473 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.243476 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.243827 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:07.244150 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:07.244443 (+ 293us) server_negotiation.cc:299] Negotiation successful
0810 11:37:07.244672 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":780,"thread_start_us":667,"threads_started":1}
W20260810 11:37:07.246063 6189 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:57098
W20260810 11:37:07.246731 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6001: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:07.246919 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 6001
I20260810 11:37:07.272814 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6198
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[5633](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361825, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:07.289948 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:07.274431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.3.240.195:39229)
0810 11:37:07.274617 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:07.274621 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:07.274638 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:37:07.274691 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:07.274695 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:07.274752 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:07.274879 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:07.276313 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.277249 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.278509 (+ 1260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:07.279256 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:07.279397 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:07.283743 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:07.283758 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:07.283782 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:07.283821 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:07.287008 (+ 3187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.287684 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.287700 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.287707 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.287884 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:07.288262 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:07.288267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:07.288271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:07.288651 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:07.288975 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:07.289301 (+ 326us) server_negotiation.cc:299] Negotiation successful
0810 11:37:07.289543 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 11:37:07.290688 6383 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:46678
W20260810 11:37:07.291420 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6198: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:07.291584 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 6198
I20260810 11:37:07.322243 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5652
I20260810 11:37:07.323563 5752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:07.443499 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5652
2026-08-10T11:37:07Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5735 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[6523](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[6523](info): set up 2 sockets
Aug 10 11:37:07 dist-test-slave-drl0 krb5kdc[6523](info): commencing operation
krb5kdc: starting...
W20260810 11:37:08.779639 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.003s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:08 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:09.059752 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36819
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36819
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:09.450764 6541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:09.451092 6541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:09.451404 6541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:09.459587 6541 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:09.459698 6541 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 11:37:09.459751 6541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:09.459882 6541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:09.459996 6541 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:09.460084 6541 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:09.470513 6541 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36819
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36819
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:09.471952 6541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:09.473945 6541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:09.488313 6548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:09.488425 6552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:09.488448 6549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:09.490423 6541 server_base.cc:1034] running on GCE node
I20260810 11:37:09.491667 6541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:09.493538 6541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:09.494865 6541 hybrid_clock.cc:648] HybridClock initialized: now 1786361829494791 us; error 52 us; skew 500 ppm
I20260810 11:37:09.495345 6541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:09 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:09.502003 6541 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:09.503832 6541 webserver.cc:460] Webserver started at http://127.3.240.254:45611/ using document root <none> and password file <none>
I20260810 11:37:09.504639 6541 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:09.504730 6541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:09.505012 6541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:09.509322 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "842ddf17916d46dabcba4dca17922fad"
format_stamp: "Formatted at 2026-08-10 11:37:09 on dist-test-slave-drl0"
server_key: "2a7d4fcd2a87614588af325a58729416"
server_key_iv: "2374ab1b3206e38c5bf62b0abdd810ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:09.510267 6541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "842ddf17916d46dabcba4dca17922fad"
format_stamp: "Formatted at 2026-08-10 11:37:09 on dist-test-slave-drl0"
server_key: "2a7d4fcd2a87614588af325a58729416"
server_key_iv: "2374ab1b3206e38c5bf62b0abdd810ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:09.517146 6541 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260810 11:37:09.521669 6561 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:09.523890 6541 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:37:09.524137 6541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "842ddf17916d46dabcba4dca17922fad"
format_stamp: "Formatted at 2026-08-10 11:37:09 on dist-test-slave-drl0"
server_key: "2a7d4fcd2a87614588af325a58729416"
server_key_iv: "2374ab1b3206e38c5bf62b0abdd810ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:09.524415 6541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:09.544385 6541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:09.549641 6541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:09.550325 6541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:09.578229 6541 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:36819
I20260810 11:37:09.578248 6647 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:36819 every 8 connection(s)
I20260810 11:37:09.579840 6541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:09.584220 6649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:09.586341 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 6541
I20260810 11:37:09.586576 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:09.587445 4035 external_mini_cluster.cc:1428] Setting key 005765e700ad4b6fa28518707258be3c
I20260810 11:37:09.600756 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Bootstrap starting.
Aug 10 11:37:09 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361828, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:09.605221 6649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:09.606590 6649 log.cc:824] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:09.609891 6649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: No bootstrap required, opened a new log
I20260810 11:37:09.614079 6653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:09.589741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35116 (local address 127.3.240.254:36819)
0810 11:37:09.590651 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:09.590667 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:09.590720 (+ 53us) server_negotiation.cc:407] Connection header received
0810 11:37:09.592701 (+ 1981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:09.592727 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:09.593165 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:09.593534 (+ 369us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:09.595546 (+ 2012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:09.596852 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:09.598785 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:09.599551 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:09.599696 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:09.605133 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:09.605142 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:09.605158 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:09.605216 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:09.609897 (+ 4681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:09.610633 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:09.610645 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:09.610655 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:09.610820 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:09.611245 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:09.611250 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:09.611255 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:09.611820 (+ 565us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:09.612154 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:09.612625 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:37:09.613049 (+ 424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":669,"thread_start_us":437,"threads_started":1}
I20260810 11:37:09.619366 6649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } }
I20260810 11:37:09.619946 6649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:09.620066 6649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 842ddf17916d46dabcba4dca17922fad, State: Initialized, Role: FOLLOWER
I20260810 11:37:09.620929 6649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } }
I20260810 11:37:09.621451 6649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:09.621649 6649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:09.621860 6649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:09.624264 6649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } }
I20260810 11:37:09.624816 6649 leader_election.cc:302] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 842ddf17916d46dabcba4dca17922fad; no voters:
I20260810 11:37:09.625502 6649 leader_election.cc:288] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:09.625658 6656 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:09.626443 6656 raft_consensus.cc:686] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [term 1 LEADER]: Becoming Leader. State: Replica: 842ddf17916d46dabcba4dca17922fad, State: Running, Role: LEADER
I20260810 11:37:09.627138 6656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } }
I20260810 11:37:09.627982 6649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:09.630530 6657 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "842ddf17916d46dabcba4dca17922fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } } }
I20260810 11:37:09.630573 6658 sys_catalog.cc:455] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 842ddf17916d46dabcba4dca17922fad. Latest consensus state: current_term: 1 leader_uuid: "842ddf17916d46dabcba4dca17922fad" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "842ddf17916d46dabcba4dca17922fad" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36819 } } }
I20260810 11:37:09.631158 6657 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:09.631227 6658 sys_catalog.cc:458] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:09.634207 6670 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:09.641264 6670 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:09.651648 6670 catalog_manager.cc:1347] Generated new cluster ID: 073b617bac834c8aa39bc81b899c06dd
I20260810 11:37:09.651870 6670 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:09.695101 6670 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:09.696774 6670 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:09.706427 6670 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Generated new TSK 0
I20260810 11:37:09.707347 6670 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:09.788041 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36819
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:10.194311 6687 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:10.194715 6687 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:10.194911 6687 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:10.204360 6687 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:10.204485 6687 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:10.204674 6687 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 11:37:10.204782 6687 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:10.204877 6687 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:10.218405 6687 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:36819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:10.219935 6687 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:10.221422 6687 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:10.237425 6698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:10.237488 6703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:10.237381 6699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:10.239429 6687 server_base.cc:1034] running on GCE node
I20260810 11:37:10.240273 6687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:10.241366 6687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:10.242623 6687 hybrid_clock.cc:648] HybridClock initialized: now 1786361830242551 us; error 64 us; skew 500 ppm
I20260810 11:37:10.243060 6687 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:10 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:10.247577 6687 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:10.249325 6687 webserver.cc:460] Webserver started at http://127.3.240.193:42251/ using document root <none> and password file <none>
I20260810 11:37:10.250061 6687 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:10.250192 6687 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:10.250626 6687 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:10.254016 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "801ca8df55e83086ec2ab6fd13c2ad4f"
server_key_iv: "f7941497d1ae086a062f3d0e7766ada4"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.254995 6687 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "801ca8df55e83086ec2ab6fd13c2ad4f"
server_key_iv: "f7941497d1ae086a062f3d0e7766ada4"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.262118 6687 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260810 11:37:10.266263 6711 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:10.268275 6687 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 11:37:10.268517 6687 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "801ca8df55e83086ec2ab6fd13c2ad4f"
server_key_iv: "f7941497d1ae086a062f3d0e7766ada4"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.268956 6687 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:10.295593 6687 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:10.303210 6687 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:10.303550 6687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:10.306862 6687 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:10.307021 6687 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 11:37:10.307150 6687 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:10.307287 6687 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:10 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:10.338455 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.307459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:38185 (local address 127.3.240.254:36819)
0810 11:37:10.314154 (+ 6695us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:10.314272 (+ 118us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:10.314305 (+ 33us) server_negotiation.cc:407] Connection header received
0810 11:37:10.314375 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:10.314379 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:10.314464 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:10.314572 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:10.317172 (+ 2600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.318175 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.321141 (+ 2966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.321887 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.322093 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.328106 (+ 6013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:10.328116 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:10.328123 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:10.328177 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:10.333380 (+ 5203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.334465 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.334473 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.334478 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.334846 (+ 368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.335633 (+ 787us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.335638 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.335642 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.336091 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:10.336276 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:10.337414 (+ 1138us) server_negotiation.cc:299] Negotiation successful
0810 11:37:10.337784 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6548,"thread_start_us":6428,"threads_started":1}
I20260810 11:37:10.338971 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.308128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.193:38185)
0810 11:37:10.309019 (+ 891us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:10.309075 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:10.311015 (+ 1940us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:10.314854 (+ 3839us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:10.314863 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:10.315846 (+ 983us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:10.316849 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.316879 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.318381 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.318389 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.320922 (+ 2533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.320963 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.322416 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.322431 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.322888 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.324142 (+ 1254us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:10.324192 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:10.327849 (+ 3657us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:10.333644 (+ 5795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.333655 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.333672 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.334247 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.335095 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.335102 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.335108 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.335495 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.336247 (+ 752us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:10.336259 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:10.336850 (+ 591us) client_negotiation.cc:769] Sending connection context
0810 11:37:10.337249 (+ 399us) client_negotiation.cc:240] Negotiation successful
0810 11:37:10.337699 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":409,"threads_started":1}
I20260810 11:37:10.365074 6724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.345617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55717 (local address 127.3.240.254:36819)
0810 11:37:10.345850 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:10.345857 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:10.346775 (+ 918us) server_negotiation.cc:407] Connection header received
0810 11:37:10.347095 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:10.347104 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:10.347212 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:10.347346 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:10.349430 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.350332 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.353454 (+ 3122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.354242 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.354416 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.356863 (+ 2447us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:10.356871 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:10.356875 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:10.356917 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:10.360737 (+ 3820us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.362131 (+ 1394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.362137 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.362141 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.362305 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.362874 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.362880 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.362884 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.363388 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:10.363611 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:10.364412 (+ 801us) server_negotiation.cc:299] Negotiation successful
0810 11:37:10.364802 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:37:10.365391 6813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.345658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.193:55717)
0810 11:37:10.346615 (+ 957us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:10.346644 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:10.346853 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:10.347405 (+ 552us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:10.347415 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:10.348251 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:10.349033 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.349082 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.350518 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.350527 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.353166 (+ 2639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.353192 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.354487 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.354500 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.354612 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.355348 (+ 736us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:10.355400 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:10.356562 (+ 1162us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:10.361051 (+ 4489us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.361060 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.361063 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.361969 (+ 906us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.362434 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.362441 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.362445 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.362588 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.363569 (+ 981us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:10.363576 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:10.364003 (+ 427us) client_negotiation.cc:769] Sending connection context
0810 11:37:10.364237 (+ 234us) client_negotiation.cc:240] Negotiation successful
0810 11:37:10.364689 (+ 452us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":696,"thread_start_us":515,"threads_started":1}
I20260810 11:37:10.375598 6687 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43833
I20260810 11:37:10.375649 6883 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43833 every 8 connection(s)
I20260810 11:37:10.377127 6687 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:10.380138 6884 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36819
I20260810 11:37:10.380405 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 6687
I20260810 11:37:10.380519 6884 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:10.380699 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:10.381918 6884 heartbeater.cc:511] Master 127.3.240.254:36819 requested a full tablet report, sending...
I20260810 11:37:10.381946 4035 external_mini_cluster.cc:1428] Setting key aa3682f57fc21aacc6009cd739e88765
I20260810 11:37:10.385037 6583 ts_manager.cc:194] Registered new tserver with Master: 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833)
I20260810 11:37:10.386788 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:38185
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:10.444057 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36819
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:10.814746 6892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:10.815062 6892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:10.815171 6892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:10.822743 6892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:10.822862 6892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:10.822974 6892 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 11:37:10.823045 6892 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:10.823109 6892 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:10.833686 6892 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:36819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:10.835070 6892 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:10.836393 6892 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:10.851836 6899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:10.852252 6900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:10.852918 6902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:10.853812 6892 server_base.cc:1034] running on GCE node
I20260810 11:37:10.854776 6892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:10.856168 6892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:10.857473 6892 hybrid_clock.cc:648] HybridClock initialized: now 1786361830857427 us; error 58 us; skew 500 ppm
I20260810 11:37:10.858071 6892 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:10 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:10.862317 6892 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:10.864575 6892 webserver.cc:460] Webserver started at http://127.3.240.194:41905/ using document root <none> and password file <none>
I20260810 11:37:10.865442 6892 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:10.865617 6892 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:10.866099 6892 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:10.869529 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "1f43ab78be084f8caaaea70d64d5ad63"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "6fd1541c358cfa8625dbb85e01b61bf2"
server_key_iv: "f628832913d123197c060de90d95fd76"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.870388 6892 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "1f43ab78be084f8caaaea70d64d5ad63"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "6fd1541c358cfa8625dbb85e01b61bf2"
server_key_iv: "f628832913d123197c060de90d95fd76"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.877272 6892 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 11:37:10.881829 6912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:10.884271 6892 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:37:10.884661 6892 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "1f43ab78be084f8caaaea70d64d5ad63"
format_stamp: "Formatted at 2026-08-10 11:37:10 on dist-test-slave-drl0"
server_key: "6fd1541c358cfa8625dbb85e01b61bf2"
server_key_iv: "f628832913d123197c060de90d95fd76"
server_key_version: "encryptionkey@0"
I20260810 11:37:10.884909 6892 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:10.908349 6892 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:10.913089 6892 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:10.913532 6892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:10.917868 6892 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:10.918035 6892 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:10.918198 6892 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:10.918320 6892 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:10 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361830, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:10.947715 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.917914 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42539 (local address 127.3.240.254:36819)
0810 11:37:10.918511 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:10.918521 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:10.919704 (+ 1183us) server_negotiation.cc:407] Connection header received
0810 11:37:10.921975 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:10.921985 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:10.922112 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:10.922310 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:10.925164 (+ 2854us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.925996 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.929298 (+ 3302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.930064 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.930203 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.937304 (+ 7101us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:10.937314 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:10.937329 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:10.937378 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:10.942458 (+ 5080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.943879 (+ 1421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.943887 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.943893 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.944053 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.944779 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.944789 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.944795 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.945286 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:10.945512 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:10.946641 (+ 1129us) server_negotiation.cc:299] Negotiation successful
0810 11:37:10.947044 (+ 403us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":309,"threads_started":1}
I20260810 11:37:10.948553 6924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.918317 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.194:42539)
0810 11:37:10.919498 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:10.919546 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:10.921631 (+ 2085us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:10.922766 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:10.922784 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:10.924028 (+ 1244us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:10.924922 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.924946 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.926163 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.926170 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.929049 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.929071 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.930603 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.930617 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.931341 (+ 724us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.932621 (+ 1280us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:10.932677 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:10.936916 (+ 4239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:10.942665 (+ 5749us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.942677 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.942700 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.943613 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.944202 (+ 589us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.944207 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.944211 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.944582 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.945506 (+ 924us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:10.945519 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:10.946068 (+ 549us) client_negotiation.cc:769] Sending connection context
0810 11:37:10.946475 (+ 407us) client_negotiation.cc:240] Negotiation successful
0810 11:37:10.947079 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":917,"thread_start_us":621,"threads_started":1}
I20260810 11:37:10.976549 7025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.957579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.194:49509)
0810 11:37:10.958258 (+ 679us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:10.958286 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:10.958443 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:10.959039 (+ 596us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:10.959050 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:10.959566 (+ 516us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:10.959857 (+ 291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.959866 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.961777 (+ 1911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.961789 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.963552 (+ 1763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:10.963577 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.964853 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.964867 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:10.965173 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.966146 (+ 973us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:10.966177 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:10.967900 (+ 1723us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:10.972990 (+ 5090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.972998 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.973007 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.973524 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.974028 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:10.974036 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:10.974041 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:10.974230 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:10.975078 (+ 848us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:10.975086 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:10.975352 (+ 266us) client_negotiation.cc:769] Sending connection context
0810 11:37:10.975577 (+ 225us) client_negotiation.cc:240] Negotiation successful
0810 11:37:10.976039 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"thread_start_us":333,"threads_started":1}
I20260810 11:37:10.977499 6921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:10.958092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49509 (local address 127.3.240.254:36819)
0810 11:37:10.958327 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:10.958336 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:10.958359 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:10.958606 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:10.958614 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:10.958723 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:10.959007 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:10.960020 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.961420 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.963777 (+ 2357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:10.964572 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:10.964734 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:10.968266 (+ 3532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:10.968274 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:10.968280 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:10.968342 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:10.972826 (+ 4484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.973676 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.973683 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.973690 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.973862 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:10.974380 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:10.974389 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:10.974394 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:10.974918 (+ 524us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:10.975121 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:10.976913 (+ 1792us) server_negotiation.cc:299] Negotiation successful
0810 11:37:10.977184 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:37:10.978356 6892 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38743
I20260810 11:37:10.978380 7086 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38743 every 8 connection(s)
I20260810 11:37:10.980000 6892 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:10.982054 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 6892
I20260810 11:37:10.982277 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:10.983035 4035 external_mini_cluster.cc:1428] Setting key 45fb7e361fa6d0ac0ff192742b9c31d8
I20260810 11:37:10.983657 7087 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36819
I20260810 11:37:10.983978 7087 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:10.984871 7087 heartbeater.cc:511] Master 127.3.240.254:36819 requested a full tablet report, sending...
I20260810 11:37:10.986678 6583 ts_manager.cc:194] Registered new tserver with Master: 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743)
I20260810 11:37:10.987586 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:42539
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:11.039719 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36819
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46727
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:11.390254 6884 heartbeater.cc:503] Master 127.3.240.254:36819 was elected leader, sending a full tablet report...
W20260810 11:37:11.448755 7092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:11.449138 7092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:11.449270 7092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:11.458583 7092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:11.458720 7092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:11.458827 7092 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260810 11:37:11.458895 7092 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:37:11.459050 7092 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:11.470490 7092 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46727
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:36819
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:11.472679 7092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:11.475062 7092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:11.490767 7104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:11.490863 7106 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:11.490720 7108 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:11.496984 7092 server_base.cc:1034] running on GCE node
I20260810 11:37:11.497700 7092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:11.498799 7092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:11.500090 7092 hybrid_clock.cc:648] HybridClock initialized: now 1786361831500010 us; error 76 us; skew 500 ppm
I20260810 11:37:11.500476 7092 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:11 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361831, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:11.503841 7092 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:11.505707 7092 webserver.cc:460] Webserver started at http://127.3.240.195:39707/ using document root <none> and password file <none>
I20260810 11:37:11.507002 7092 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:11.507225 7092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:11.507587 7092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:11.511590 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "02e57c0e7e424b8f9290163ce002facb"
format_stamp: "Formatted at 2026-08-10 11:37:11 on dist-test-slave-drl0"
server_key: "349484c83d1418de612bd21404f11bd0"
server_key_iv: "aa23f4a9abecc80231497bad28e5855e"
server_key_version: "encryptionkey@0"
I20260810 11:37:11.512535 7092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "02e57c0e7e424b8f9290163ce002facb"
format_stamp: "Formatted at 2026-08-10 11:37:11 on dist-test-slave-drl0"
server_key: "349484c83d1418de612bd21404f11bd0"
server_key_iv: "aa23f4a9abecc80231497bad28e5855e"
server_key_version: "encryptionkey@0"
I20260810 11:37:11.519673 7092 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 11:37:11.524081 7117 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:11.525916 7092 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:37:11.526110 7092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "02e57c0e7e424b8f9290163ce002facb"
format_stamp: "Formatted at 2026-08-10 11:37:11 on dist-test-slave-drl0"
server_key: "349484c83d1418de612bd21404f11bd0"
server_key_iv: "aa23f4a9abecc80231497bad28e5855e"
server_key_version: "encryptionkey@0"
I20260810 11:37:11.526583 7092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:11.556730 7092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:11.561694 7092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:11.562048 7092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:11.565218 7092 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:11.565322 7092 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:11.565466 7092 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:11.565574 7092 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:37:11 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361831, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:11.595314 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.565624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:49909 (local address 127.3.240.254:36819)
0810 11:37:11.566112 (+ 488us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.566119 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.568190 (+ 2071us) server_negotiation.cc:407] Connection header received
0810 11:37:11.571167 (+ 2977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.571176 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.571299 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.571449 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:11.573748 (+ 2299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.574979 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.578580 (+ 3601us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.579401 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.579600 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.584453 (+ 4853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:11.584461 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:11.584466 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:11.584521 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:11.588887 (+ 4366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.590849 (+ 1962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.590884 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.590892 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.591195 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.592547 (+ 1352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.592557 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.592562 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.593159 (+ 597us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:11.593338 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.594457 (+ 1119us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.594799 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":255,"threads_started":1}
I20260810 11:37:11.597157 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.566230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.195:49909)
0810 11:37:11.567978 (+ 1748us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.568027 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.570655 (+ 2628us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.571716 (+ 1061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.571724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.572288 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:11.573509 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.573543 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.575175 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.575180 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.578341 (+ 3161us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.578363 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.579730 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.579738 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.579997 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.580746 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:11.580788 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:11.584107 (+ 3319us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:11.589091 (+ 4984us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.589100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.589119 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.589965 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.591710 (+ 1745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.591720 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.591728 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.592299 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.593460 (+ 1161us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:11.593468 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:11.593978 (+ 510us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.594360 (+ 382us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.595167 (+ 807us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1332,"thread_start_us":755,"threads_started":1}
I20260810 11:37:11.624866 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.603333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:51355 (local address 127.3.240.254:36819)
0810 11:37:11.603648 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.603657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.607768 (+ 4111us) server_negotiation.cc:407] Connection header received
0810 11:37:11.608053 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.608059 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.608135 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.608247 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:11.609257 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.609993 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.611865 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.612644 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.612795 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.616746 (+ 3951us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:11.616755 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:11.616759 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:11.616814 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:11.620970 (+ 4156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.622118 (+ 1148us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.622122 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.622125 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.622270 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.622852 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.622858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.622861 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.623541 (+ 680us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:11.623731 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.624428 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.624614 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:37:11.625526 7225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.603495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.3.240.195:51355)
0810 11:37:11.607613 (+ 4118us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.607639 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.607848 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.608276 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.608281 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.608809 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:11.609110 (+ 301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.609120 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.610143 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.610149 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.611564 (+ 1415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.611585 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.613068 (+ 1483us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.613080 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.613322 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.614292 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:11.614323 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:11.616276 (+ 1953us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:11.621171 (+ 4895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.621180 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.621191 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.621924 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.622392 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.622400 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.622406 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.622602 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.623729 (+ 1127us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:11.623740 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:11.624043 (+ 303us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.624276 (+ 233us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.624777 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3896,"thread_start_us":3689,"threads_started":1}
I20260810 11:37:11.638266 7092 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40641
I20260810 11:37:11.638365 7282 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40641 every 8 connection(s)
I20260810 11:37:11.639724 7092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:11.642335 7285 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36819
I20260810 11:37:11.642911 7285 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:11.643993 7285 heartbeater.cc:511] Master 127.3.240.254:36819 requested a full tablet report, sending...
I20260810 11:37:11.645727 6583 ts_manager.cc:194] Registered new tserver with Master: 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:11.646634 6583 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:49909
I20260810 11:37:11.649446 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 7092
I20260810 11:37:11.649592 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:11.650053 4035 external_mini_cluster.cc:1428] Setting key 1ebeaee2173e32f44b01f83e2edb31fa
I20260810 11:37:11.653692 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 11:37:11 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 11:37:11 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361831, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:11.713950 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.697204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35124 (local address 127.3.240.254:36819)
0810 11:37:11.697407 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.697411 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.697858 (+ 447us) server_negotiation.cc:407] Connection header received
0810 11:37:11.698103 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.698108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.698212 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.698336 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:11.700261 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.701441 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.702714 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.703239 (+ 525us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.703382 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.707592 (+ 4210us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:11.707597 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:11.707600 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:11.707635 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:11.710722 (+ 3087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.711751 (+ 1029us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.711760 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.711763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.711917 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.712332 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.712341 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.712344 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.712659 (+ 315us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:11.712982 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.713301 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.713556 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:37:11.719842 6583 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:11.728842 6583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:11.759438 7305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.744519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38743 (local address 127.0.0.1:34114)
0810 11:37:11.747411 (+ 2892us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.747432 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.747633 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.748608 (+ 975us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.748617 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.748667 (+ 50us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:11.749373 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.749398 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.751987 (+ 2589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.751993 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.756560 (+ 4567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.756579 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.758246 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.758258 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.758505 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.758617 (+ 112us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.758743 (+ 126us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.758972 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2757,"mutex_wait_us":111,"thread_start_us":2857,"threads_started":1}
I20260810 11:37:11.759769 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.745162 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.3.240.194:38743)
0810 11:37:11.745715 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.745727 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.747521 (+ 1794us) server_negotiation.cc:407] Connection header received
0810 11:37:11.747861 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.747872 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.748151 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.748322 (+ 171us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:11.749907 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.751786 (+ 1879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.756737 (+ 4951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.757976 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.758861 (+ 885us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.759102 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.759174 (+ 72us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.759355 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":261,"threads_started":1}
I20260810 11:37:11.760154 7302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.744123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40641 (local address 127.0.0.1:47742)
0810 11:37:11.744988 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.745037 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.745243 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.748770 (+ 3527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.748777 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.748824 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:11.749800 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.749823 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.754129 (+ 4306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.754138 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.757262 (+ 3124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.757301 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.759339 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.759346 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.759530 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.759563 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.759652 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.759790 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"mutex_wait_us":34,"thread_start_us":392,"threads_started":1}
I20260810 11:37:11.760598 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.744706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47742 (local address 127.3.240.195:40641)
0810 11:37:11.747833 (+ 3127us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.747845 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.747904 (+ 59us) server_negotiation.cc:407] Connection header received
0810 11:37:11.748053 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.748095 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.748461 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.750046 (+ 1585us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:11.750550 (+ 504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.753868 (+ 3318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.757455 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.758723 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.758886 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.759182 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.760087 (+ 905us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.760249 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2698,"thread_start_us":2488,"threads_started":1}
I20260810 11:37:11.765221 7310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.748758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43833 (local address 127.0.0.1:53432)
0810 11:37:11.757188 (+ 8430us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.757211 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.757354 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.758079 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.758085 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.758119 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:11.758650 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.758661 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.760763 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.760768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.762504 (+ 1736us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.762517 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.764408 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.764414 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.764581 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.764613 (+ 32us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.764716 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.764854 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8297,"mutex_wait_us":53,"thread_start_us":8144,"threads_started":1}
I20260810 11:37:11.765242 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.748690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53432 (local address 127.3.240.193:43833)
0810 11:37:11.756401 (+ 7711us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.756411 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.757336 (+ 925us) server_negotiation.cc:407] Connection header received
0810 11:37:11.757589 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.757597 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.757858 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.758073 (+ 215us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:11.758819 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.760346 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.762661 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.764177 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.764335 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.764556 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.764783 (+ 227us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.764946 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7506,"thread_start_us":7335,"threads_started":1}
I20260810 11:37:11.764815 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 25d5e69e1e9247228a3d668e8c829589 (DEFAULT_TABLE table=test-table [id=9ae8de1f68d64b3b87339e26bc435be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:11.765249 6987 tablet_service.cc:1467] Processing CreateTablet for tablet 25d5e69e1e9247228a3d668e8c829589 (DEFAULT_TABLE table=test-table [id=9ae8de1f68d64b3b87339e26bc435be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:11.766721 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d5e69e1e9247228a3d668e8c829589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:11.766569 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d5e69e1e9247228a3d668e8c829589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:11.769095 6786 tablet_service.cc:1467] Processing CreateTablet for tablet 25d5e69e1e9247228a3d668e8c829589 (DEFAULT_TABLE table=test-table [id=9ae8de1f68d64b3b87339e26bc435be1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:11.770432 6786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 25d5e69e1e9247228a3d668e8c829589. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:11.779572 7317 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Bootstrap starting.
I20260810 11:37:11.781965 7318 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Bootstrap starting.
I20260810 11:37:11.785192 7317 tablet_bootstrap.cc:654] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:11.785419 7319 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Bootstrap starting.
I20260810 11:37:11.787171 7318 tablet_bootstrap.cc:654] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:11.787925 7317 log.cc:824] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:11.788901 7318 log.cc:824] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:11.789196 7319 tablet_bootstrap.cc:654] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:11.791009 7319 log.cc:824] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:11.791392 7317 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: No bootstrap required, opened a new log
I20260810 11:37:11.791692 7317 ts_tablet_manager.cc:1397] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.004s
I20260810 11:37:11.794780 7319 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: No bootstrap required, opened a new log
I20260810 11:37:11.795208 7319 ts_tablet_manager.cc:1397] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.005s
I20260810 11:37:11.801158 7317 raft_consensus.cc:348] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.801486 7318 tablet_bootstrap.cc:492] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: No bootstrap required, opened a new log
I20260810 11:37:11.801550 7317 raft_consensus.cc:374] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:11.801635 7317 raft_consensus.cc:729] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Initialized, Role: FOLLOWER
I20260810 11:37:11.801769 7318 ts_tablet_manager.cc:1397] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.005s
I20260810 11:37:11.802311 7317 consensus_queue.cc:260] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.803427 7285 heartbeater.cc:503] Master 127.3.240.254:36819 was elected leader, sending a full tablet report...
I20260810 11:37:11.805457 7317 ts_tablet_manager.cc:1428] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Time spent starting tablet: real 0.014s user 0.012s sys 0.000s
I20260810 11:37:11.805130 7319 raft_consensus.cc:348] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.805954 7319 raft_consensus.cc:374] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:11.806219 7319 raft_consensus.cc:729] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Initialized, Role: FOLLOWER
I20260810 11:37:11.807251 7319 consensus_queue.cc:260] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.809659 7319 ts_tablet_manager.cc:1428] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 11:37:11.814553 7318 raft_consensus.cc:348] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.814973 7318 raft_consensus.cc:374] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:11.815074 7318 raft_consensus.cc:729] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f43ab78be084f8caaaea70d64d5ad63, State: Initialized, Role: FOLLOWER
I20260810 11:37:11.815611 7318 consensus_queue.cc:260] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.816577 7087 heartbeater.cc:503] Master 127.3.240.254:36819 was elected leader, sending a full tablet report...
I20260810 11:37:11.817103 7318 ts_tablet_manager.cc:1428] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent starting tablet: real 0.015s user 0.011s sys 0.003s
W20260810 11:37:11.880388 6885 tablet.cc:2369] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:11.891516 7286 tablet.cc:2369] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:11.905180 7327 raft_consensus.cc:482] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:11.905505 7327 raft_consensus.cc:504] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.907402 7327 leader_election.cc:288] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833)
I20260810 11:37:11.914613 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.908320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:58613 (local address 127.3.240.194:38743)
0810 11:37:11.908609 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.908616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.908638 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:37:11.908911 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.908921 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.909059 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.909196 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:11.909827 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.910651 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.912542 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.913418 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.913762 (+ 344us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.913898 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.914177 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.914350 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 11:37:11.914654 7133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.908058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38743 (local address 127.3.240.195:58613)
0810 11:37:11.908485 (+ 427us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.908510 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.908669 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.909376 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.909382 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.909424 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:11.909656 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.909666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.910833 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.910839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.912353 (+ 1514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.912368 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.913709 (+ 1341us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.913731 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.913989 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.914016 (+ 27us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.914144 (+ 128us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.914296 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":201,"mutex_wait_us":1}
I20260810 11:37:11.915109 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.909016 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38295 (local address 127.3.240.193:43833)
0810 11:37:11.909253 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.909259 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.909491 (+ 232us) server_negotiation.cc:407] Connection header received
0810 11:37:11.909710 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.909714 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.909772 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.909872 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:11.910449 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.911397 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.913174 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.914124 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.914339 (+ 215us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.914427 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.914706 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.914829 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 11:37:11.915158 7334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.908636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43833 (local address 127.3.240.195:38295)
0810 11:37:11.909378 (+ 742us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.909398 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.909558 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.909893 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.909899 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.909924 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:11.910277 (+ 353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.910287 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.911555 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.911561 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.913015 (+ 1454us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.913027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.914391 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.914398 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.914545 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.914567 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.914681 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.914802 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":399,"threads_started":1}
I20260810 11:37:11.915912 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d5e69e1e9247228a3d668e8c829589" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63" is_pre_election: true
I20260810 11:37:11.916306 6825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d5e69e1e9247228a3d668e8c829589" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" is_pre_election: true
I20260810 11:37:11.916450 7024 raft_consensus.cc:2454] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 0.
I20260810 11:37:11.916814 6825 raft_consensus.cc:2454] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 0.
I20260810 11:37:11.917218 7122 leader_election.cc:302] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 1f43ab78be084f8caaaea70d64d5ad63; no voters:
I20260810 11:37:11.917618 7327 raft_consensus.cc:2781] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:11.917798 7327 raft_consensus.cc:482] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:11.917999 7327 raft_consensus.cc:3037] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:11.920903 7327 raft_consensus.cc:504] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.921836 7327 leader_election.cc:288] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 election: Requested vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833)
I20260810 11:37:11.922663 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d5e69e1e9247228a3d668e8c829589" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63"
I20260810 11:37:11.922663 6825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "25d5e69e1e9247228a3d668e8c829589" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab"
I20260810 11:37:11.923045 6825 raft_consensus.cc:3037] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:11.923045 7024 raft_consensus.cc:3037] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:11.927120 7024 raft_consensus.cc:2454] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 1.
I20260810 11:37:11.927120 6825 raft_consensus.cc:2454] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 1.
I20260810 11:37:11.927567 7122 leader_election.cc:302] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:11.927968 7327 raft_consensus.cc:2781] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:11.928507 7327 raft_consensus.cc:686] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 1 LEADER]: Becoming Leader. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Running, Role: LEADER
I20260810 11:37:11.929180 7327 consensus_queue.cc:237] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } }
I20260810 11:37:11.936053 6582 catalog_manager.cc:5495] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e57c0e7e424b8f9290163ce002facb (127.3.240.195). New cstate: current_term: 1 leader_uuid: "02e57c0e7e424b8f9290163ce002facb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } health_report { overall_health: UNKNOWN } } }
Aug 10 11:37:11 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361829, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
W20260810 11:37:11.984519 7088 tablet.cc:2369] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:11.987597 7130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.970365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35134 (local address 127.3.240.254:36819)
0810 11:37:11.970727 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:11.970736 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:11.971722 (+ 986us) server_negotiation.cc:407] Connection header received
0810 11:37:11.971944 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:11.971949 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:11.972023 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:11.972126 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:11.973555 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.974477 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.975994 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.976617 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.976730 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.980769 (+ 4039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:11.980773 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:11.980777 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:11.980805 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:11.984616 (+ 3811us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.985564 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.985569 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.985572 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.985692 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:11.986093 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:11.986098 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:11.986101 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:11.986375 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:11.986532 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:11.986965 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 11:37:11.987290 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":118,"server-negotiator.queue_time_us":110}
I20260810 11:37:11.987632 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:11.970370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36819 (local address 127.0.0.1:35134)
0810 11:37:11.971536 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:11.971560 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:11.971756 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:11.972229 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:11.972234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:11.973116 (+ 882us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:11.973410 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.973420 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.974715 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.974720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.975825 (+ 1105us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:11.975838 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:11.976844 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:11.976850 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:11.976994 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:11.977734 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:11.977760 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:11.980575 (+ 2815us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:11.984783 (+ 4208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.984795 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.984802 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.985434 (+ 632us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.985799 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:11.985806 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:11.985813 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:11.985978 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:11.986504 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:11.986512 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:11.986672 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 11:37:11.986837 (+ 165us) client_negotiation.cc:240] Negotiation successful
0810 11:37:11.987157 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":716,"threads_started":1}
I20260810 11:37:11.993510 6583 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35134:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 11:37:11.994680 6583 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:12.003263 6987 tablet_service.cc:1467] Processing CreateTablet for tablet 010f4e6ed9e1462c8da1b10366ced642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa210dedc1b4427860728d333e14bc5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:12.003261 6786 tablet_service.cc:1467] Processing CreateTablet for tablet 010f4e6ed9e1462c8da1b10366ced642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa210dedc1b4427860728d333e14bc5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:12.003473 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 010f4e6ed9e1462c8da1b10366ced642 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=faa210dedc1b4427860728d333e14bc5]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:37:12.004175 6987 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010f4e6ed9e1462c8da1b10366ced642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.004223 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010f4e6ed9e1462c8da1b10366ced642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.004222 6786 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 010f4e6ed9e1462c8da1b10366ced642. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.010015 7318 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: Bootstrap starting.
I20260810 11:37:12.011821 7318 tablet_bootstrap.cc:654] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.012683 7319 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Bootstrap starting.
I20260810 11:37:12.014446 7317 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: Bootstrap starting.
I20260810 11:37:12.014496 7319 tablet_bootstrap.cc:654] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.014987 7318 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: No bootstrap required, opened a new log
I20260810 11:37:12.015166 7318 ts_tablet_manager.cc:1397] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260810 11:37:12.016067 7317 tablet_bootstrap.cc:654] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.016162 7318 raft_consensus.cc:348] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.016474 7318 raft_consensus.cc:374] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.016594 7318 raft_consensus.cc:729] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f43ab78be084f8caaaea70d64d5ad63, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.016870 7318 consensus_queue.cc:260] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.017108 7319 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: No bootstrap required, opened a new log
I20260810 11:37:12.017261 7319 ts_tablet_manager.cc:1397] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.003s
I20260810 11:37:12.017840 7333 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.018260 7333 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.018184 7319 raft_consensus.cc:348] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.018471 7319 raft_consensus.cc:374] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.018539 7317 tablet_bootstrap.cc:492] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: No bootstrap required, opened a new log
I20260810 11:37:12.018582 7319 raft_consensus.cc:729] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.018677 7317 ts_tablet_manager.cc:1397] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:12.018841 7319 consensus_queue.cc:260] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.019568 7319 ts_tablet_manager.cc:1428] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260810 11:37:12.019526 7317 raft_consensus.cc:348] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.019775 7317 raft_consensus.cc:374] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.019479 7332 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.019881 7317 raft_consensus.cc:729] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.019934 7332 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.019949 7318 ts_tablet_manager.cc:1428] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260810 11:37:12.020112 7317 consensus_queue.cc:260] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.020494 7327 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.020696 7317 ts_tablet_manager.cc:1428] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 11:37:12.020787 7327 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.256397 7332 raft_consensus.cc:482] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:12.256796 7332 raft_consensus.cc:504] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.259012 7332 leader_election.cc:288] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:12.267021 7349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.259554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38743 (local address 127.3.240.193:55847)
0810 11:37:12.260698 (+ 1144us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.260713 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.260964 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.261377 (+ 413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.261380 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.261415 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:12.261752 (+ 337us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.261761 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.263190 (+ 1429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.263193 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.264739 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.264752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.266173 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.266181 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.266397 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.266435 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.266521 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.266618 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":939,"mutex_wait_us":1,"thread_start_us":637,"threads_started":1}
I20260810 11:37:12.267221 7309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.259529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55847 (local address 127.3.240.194:38743)
0810 11:37:12.259795 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.259804 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.260825 (+ 1021us) server_negotiation.cc:407] Connection header received
0810 11:37:12.261125 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.261129 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.261226 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.261336 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:12.261930 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.263028 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.264909 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.265925 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.266056 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.266109 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.266702 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.266853 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:37:12.267522 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.259768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:41905 (local address 127.3.240.195:40641)
0810 11:37:12.260037 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.260046 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.260775 (+ 729us) server_negotiation.cc:407] Connection header received
0810 11:37:12.261038 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.261047 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.261128 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.261196 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:12.261928 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.262613 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.265235 (+ 2622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.266074 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.266258 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.266337 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.267127 (+ 790us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.267273 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 11:37:12.268146 7350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.259488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40641 (local address 127.3.240.193:41905)
0810 11:37:12.260607 (+ 1119us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.260652 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.260867 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.261255 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.261263 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.261294 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:12.261746 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.261756 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.262770 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.262776 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.265011 (+ 2235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.265028 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.266524 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.266534 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.266870 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.266913 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.267081 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.267786 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":981,"mutex_wait_us":139,"thread_start_us":863,"threads_started":1}
I20260810 11:37:12.268147 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010f4e6ed9e1462c8da1b10366ced642" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63" is_pre_election: true
I20260810 11:37:12.268704 7024 raft_consensus.cc:2454] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:12.269619 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010f4e6ed9e1462c8da1b10366ced642" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb" is_pre_election: true
I20260810 11:37:12.270138 7220 raft_consensus.cc:2454] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:12.270916 6716 leader_election.cc:302] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:12.271531 7332 raft_consensus.cc:2781] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:12.271688 7332 raft_consensus.cc:482] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:12.271797 7332 raft_consensus.cc:3037] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.274488 7332 raft_consensus.cc:504] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.275218 7332 leader_election.cc:288] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Requested vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:12.275736 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010f4e6ed9e1462c8da1b10366ced642" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63"
I20260810 11:37:12.276001 7024 raft_consensus.cc:3037] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.276026 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "010f4e6ed9e1462c8da1b10366ced642" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb"
I20260810 11:37:12.276328 7220 raft_consensus.cc:3037] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.278108 7024 raft_consensus.cc:2454] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:12.278587 6715 leader_election.cc:302] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f43ab78be084f8caaaea70d64d5ad63, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:12.279044 7332 raft_consensus.cc:2781] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:12.279433 7220 raft_consensus.cc:2454] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:12.279544 7332 raft_consensus.cc:686] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Running, Role: LEADER
I20260810 11:37:12.280371 7332 consensus_queue.cc:237] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.284335 7351 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: TxnStatusTablet state changed. Reason: New leader 35ae26b2a6f748ef89a4fec8f7f7d5ab. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.284761 7351 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:12.285636 7353 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 11:37:12.285991 7353 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 11:37:12.286983 6584 catalog_manager.cc:5495] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193). New cstate: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } health_report { overall_health: UNKNOWN } } }
I20260810 11:37:12.329083 7344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.322018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43833 (local address 127.0.0.1:53434)
0810 11:37:12.322278 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.322297 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.322425 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.323033 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.323036 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.323047 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:12.323219 (+ 172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.323227 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.324770 (+ 1543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.324774 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.325809 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.325821 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.326860 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.326870 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.327168 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.327251 (+ 83us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.328526 (+ 1275us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.328549 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.328638 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.328766 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":62}
I20260810 11:37:12.329777 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.322150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53434 (local address 127.3.240.193:43833)
0810 11:37:12.322327 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.322332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.322350 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:12.322640 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.322646 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.322775 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.322953 (+ 178us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.323363 (+ 410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.324583 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.325964 (+ 1381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.326634 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.326780 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.327530 (+ 750us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.328366 (+ 836us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.328515 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.328835 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.329005 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 11:37:12.337672 7024 raft_consensus.cc:1261] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:12.337772 7220 raft_consensus.cc:1261] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:12.338654 7351 consensus_queue.cc:1035] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.338997 7332 consensus_queue.cc:1035] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.351394 7335 consensus_queue.cc:1035] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.362124 7327 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: TxnStatusTablet state changed. Reason: New leader 35ae26b2a6f748ef89a4fec8f7f7d5ab. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.362565 7327 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.366168 7364 consensus_queue.cc:1035] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.368247 7333 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: TxnStatusTablet state changed. Reason: New leader 35ae26b2a6f748ef89a4fec8f7f7d5ab. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.368693 7333 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.378170 7361 mvcc.cc:205] Tried to move back new op lower bound from 7316938065245646848 to 7316938065024626688. Current Snapshot: MvccSnapshot[applied={T|T < 7316938065245646848}]
I20260810 11:37:12.380767 7333 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.381608 7333 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 1f43ab78be084f8caaaea70d64d5ad63: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.384326 7327 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.384749 7327 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 02e57c0e7e424b8f9290163ce002facb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:37:12.416222 7351 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.416530 7351 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:12.416788 7373 tablet_replica.cc:439] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } } }
I20260810 11:37:12.417229 7373 tablet_replica.cc:442] T 010f4e6ed9e1462c8da1b10366ced642 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:37:12.434744 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.422113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47746 (local address 127.3.240.195:40641)
0810 11:37:12.422279 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.422286 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.427600 (+ 5314us) server_negotiation.cc:407] Connection header received
0810 11:37:12.427839 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.427845 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.427970 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.428140 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.429227 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.430391 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.432079 (+ 1688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.432737 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.432883 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.433296 (+ 413us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.434063 (+ 767us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.434211 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.434354 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.434455 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 11:37:12.454001 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.446735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46009 (local address 127.3.240.193:43833)
0810 11:37:12.447130 (+ 395us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.447136 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.447232 (+ 96us) server_negotiation.cc:407] Connection header received
0810 11:37:12.447437 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.447441 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.447518 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.447628 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.448102 (+ 474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.449200 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.451536 (+ 2336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.452232 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.452435 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.453035 (+ 600us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.453208 (+ 173us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.453301 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.453541 (+ 240us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.453728 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237}
I20260810 11:37:12.454028 7380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.446348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43833 (local address 127.3.240.195:46009)
0810 11:37:12.447118 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.447138 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.447278 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.447651 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.447656 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.447667 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:12.447934 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.447943 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.449516 (+ 1573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.449523 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.451331 (+ 1808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.451348 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.452452 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.452459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.452638 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.452714 (+ 76us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.453339 (+ 625us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.453353 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.453449 (+ 96us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.453585 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":338,"threads_started":1}
I20260810 11:37:12.501477 7307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.494602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51657 (local address 127.3.240.195:40641)
0810 11:37:12.494854 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.494861 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.495566 (+ 705us) server_negotiation.cc:407] Connection header received
0810 11:37:12.495789 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.495795 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.495895 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.495995 (+ 100us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.496712 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.497615 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.499375 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.499917 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.500032 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.500700 (+ 668us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.500802 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.500910 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.501151 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.501250 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 11:37:12.502103 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.494484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40641 (local address 127.3.240.193:51657)
0810 11:37:12.495445 (+ 961us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.495469 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.495635 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.496027 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.496036 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.496050 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:12.496422 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.496430 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.497748 (+ 1318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.497755 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.499162 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.499175 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.500237 (+ 1062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.500245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.500456 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.500515 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.500982 (+ 467us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.500995 (+ 13us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.501675 (+ 680us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.501787 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":531,"threads_started":1}
I20260810 11:37:12.544978 6582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:12.545302 6582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:12.551568 6582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Sending DeleteTablet for 3 replicas of tablet 25d5e69e1e9247228a3d668e8c829589
I20260810 11:37:12.552842 6987 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d5e69e1e9247228a3d668e8c829589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:12 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 11:37:12.553054 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d5e69e1e9247228a3d668e8c829589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:12 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47742
I20260810 11:37:12.556466 6582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:12.557323 6582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:12.564287 7387 tablet_replica.cc:330] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: stopping tablet replica
I20260810 11:37:12.565176 7387 raft_consensus.cc:2229] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:12.566434 7387 raft_consensus.cc:2258] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:12.568965 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 (DEFAULT_TABLE table=test-table [id=4e71ec56d818457fb0aee5542012a0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:12.569798 6986 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 (DEFAULT_TABLE table=test-table [id=4e71ec56d818457fb0aee5542012a0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:12.570381 6785 tablet_service.cc:1467] Processing CreateTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 (DEFAULT_TABLE table=test-table [id=4e71ec56d818457fb0aee5542012a0a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:12.570725 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6b5984965249eeb75c0dea6c743f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.571229 6785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6b5984965249eeb75c0dea6c743f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.571619 6786 tablet_service.cc:1514] Processing DeleteTablet for tablet 25d5e69e1e9247228a3d668e8c829589 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:12 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53432
I20260810 11:37:12.574537 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2f6b5984965249eeb75c0dea6c743f85. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:12.584828 7390 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Bootstrap starting.
I20260810 11:37:12.586522 7387 ts_tablet_manager.cc:1905] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:12.586722 7390 tablet_bootstrap.cc:654] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.590620 7386 tablet_replica.cc:330] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: stopping tablet replica
I20260810 11:37:12.591290 7386 raft_consensus.cc:2229] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:12.591886 7386 raft_consensus.cc:2258] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:12.594441 7392 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Bootstrap starting.
I20260810 11:37:12.594873 7386 ts_tablet_manager.cc:1905] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:12.600435 7392 tablet_bootstrap.cc:654] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.603296 7387 ts_tablet_manager.cc:1918] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:12.603652 7387 log.cc:1196] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/25d5e69e1e9247228a3d668e8c829589
I20260810 11:37:12.604622 7387 ts_tablet_manager.cc:1939] T 25d5e69e1e9247228a3d668e8c829589 P 02e57c0e7e424b8f9290163ce002facb: Deleting consensus metadata
I20260810 11:37:12.606642 6565 catalog_manager.cc:4841] TS 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641): tablet 25d5e69e1e9247228a3d668e8c829589 (table test-table [id=9ae8de1f68d64b3b87339e26bc435be1]) successfully deleted
I20260810 11:37:12.609108 7390 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: No bootstrap required, opened a new log
I20260810 11:37:12.609263 7390 ts_tablet_manager.cc:1397] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Time spent bootstrapping tablet: real 0.025s user 0.000s sys 0.006s
I20260810 11:37:12.610069 7390 raft_consensus.cc:348] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.610339 7390 raft_consensus.cc:374] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.610450 7390 raft_consensus.cc:729] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.611609 7392 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: No bootstrap required, opened a new log
I20260810 11:37:12.611857 7392 ts_tablet_manager.cc:1397] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent bootstrapping tablet: real 0.018s user 0.006s sys 0.000s
I20260810 11:37:12.611763 7390 consensus_queue.cc:260] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.612862 7392 raft_consensus.cc:348] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.613147 7392 raft_consensus.cc:374] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.613257 7392 raft_consensus.cc:729] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.613616 7392 consensus_queue.cc:260] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.615556 7390 ts_tablet_manager.cc:1428] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260810 11:37:12.615780 7392 ts_tablet_manager.cc:1428] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:12.616618 7388 tablet_replica.cc:330] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: stopping tablet replica
I20260810 11:37:12.617129 7388 raft_consensus.cc:2229] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:12.617561 7388 raft_consensus.cc:2258] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:12.618907 7395 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Bootstrap starting.
I20260810 11:37:12.619544 7386 ts_tablet_manager.cc:1918] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:12.619731 7386 log.cc:1196] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/25d5e69e1e9247228a3d668e8c829589
I20260810 11:37:12.620350 7386 ts_tablet_manager.cc:1939] T 25d5e69e1e9247228a3d668e8c829589 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting consensus metadata
I20260810 11:37:12.621001 7395 tablet_bootstrap.cc:654] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:12.622184 6564 catalog_manager.cc:4841] TS 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743): tablet 25d5e69e1e9247228a3d668e8c829589 (table test-table [id=9ae8de1f68d64b3b87339e26bc435be1]) successfully deleted
I20260810 11:37:12.624804 7388 ts_tablet_manager.cc:1905] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:12.629480 7395 tablet_bootstrap.cc:492] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: No bootstrap required, opened a new log
I20260810 11:37:12.629701 7395 ts_tablet_manager.cc:1397] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260810 11:37:12.630821 7395 raft_consensus.cc:348] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.631151 7395 raft_consensus.cc:374] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:12.631245 7395 raft_consensus.cc:729] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f43ab78be084f8caaaea70d64d5ad63, State: Initialized, Role: FOLLOWER
I20260810 11:37:12.631852 7395 consensus_queue.cc:260] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.639060 7395 ts_tablet_manager.cc:1428] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent starting tablet: real 0.009s user 0.004s sys 0.000s
I20260810 11:37:12.648012 7388 ts_tablet_manager.cc:1918] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:12.648236 7388 log.cc:1196] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/25d5e69e1e9247228a3d668e8c829589
I20260810 11:37:12.649322 7388 ts_tablet_manager.cc:1939] T 25d5e69e1e9247228a3d668e8c829589 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting consensus metadata
I20260810 11:37:12.654063 6564 catalog_manager.cc:4841] TS 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833): tablet 25d5e69e1e9247228a3d668e8c829589 (table test-table [id=9ae8de1f68d64b3b87339e26bc435be1]) successfully deleted
I20260810 11:37:12.765719 7332 raft_consensus.cc:482] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:12.766076 7332 raft_consensus.cc:504] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.767941 7332 leader_election.cc:288] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:12.768486 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6b5984965249eeb75c0dea6c743f85" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63" is_pre_election: true
I20260810 11:37:12.768702 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6b5984965249eeb75c0dea6c743f85" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb" is_pre_election: true
I20260810 11:37:12.768971 7022 raft_consensus.cc:2454] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:12.769194 7220 raft_consensus.cc:2454] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:12.769879 6716 leader_election.cc:302] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:12.770258 7332 raft_consensus.cc:2781] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:12.770452 7332 raft_consensus.cc:482] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:12.770573 7332 raft_consensus.cc:3037] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.773895 7332 raft_consensus.cc:504] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.775177 7332 leader_election.cc:288] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Requested vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:12.775689 7220 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6b5984965249eeb75c0dea6c743f85" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb"
I20260810 11:37:12.775936 7220 raft_consensus.cc:3037] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.777995 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2f6b5984965249eeb75c0dea6c743f85" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63"
I20260810 11:37:12.778656 7022 raft_consensus.cc:3037] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:12.779758 7220 raft_consensus.cc:2454] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:12.780596 6716 leader_election.cc:302] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:12.781273 7332 raft_consensus.cc:2781] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:12.781523 7332 raft_consensus.cc:686] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Running, Role: LEADER
I20260810 11:37:12.782145 7332 consensus_queue.cc:237] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:12.783387 7022 raft_consensus.cc:2454] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:12.789178 6582 catalog_manager.cc:5495] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193). New cstate: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } health_report { overall_health: UNKNOWN } } }
I20260810 11:37:12.861374 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.850192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53450 (local address 127.3.240.193:43833)
0810 11:37:12.850433 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.850440 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.851043 (+ 603us) server_negotiation.cc:407] Connection header received
0810 11:37:12.851626 (+ 583us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.851654 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.851825 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.851992 (+ 167us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.852832 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.854452 (+ 1620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.857172 (+ 2720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.858182 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.858404 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.859398 (+ 994us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.859761 (+ 363us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.859956 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.860186 (+ 230us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.860292 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 11:37:12.890578 7382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.879102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43833 (local address 127.3.240.193:52467)
0810 11:37:12.879385 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:12.879416 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:12.879613 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:12.880174 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:12.880181 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:12.880199 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:37:12.883214 (+ 3015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.883233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.885420 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.885425 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.887425 (+ 2000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:12.887441 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.888712 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.888719 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:12.889076 (+ 357us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.889203 (+ 127us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.889812 (+ 609us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.889824 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 11:37:12.889878 (+ 54us) client_negotiation.cc:240] Negotiation successful
0810 11:37:12.890035 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":2602}
I20260810 11:37:12.891443 7312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:12.878814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52467 (local address 127.3.240.193:43833)
0810 11:37:12.879110 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:12.879118 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:12.879767 (+ 649us) server_negotiation.cc:407] Connection header received
0810 11:37:12.879846 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:12.879851 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:12.879965 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:12.880073 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:12.883449 (+ 3376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.884872 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.887599 (+ 2727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:12.888493 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:12.888616 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:12.889446 (+ 830us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:12.889653 (+ 207us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:12.890848 (+ 1195us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:12.890913 (+ 65us) server_negotiation.cc:299] Negotiation successful
0810 11:37:12.891014 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":127}
I20260810 11:37:12.904754 7216 raft_consensus.cc:1261] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:12.905344 7024 raft_consensus.cc:1261] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:12.906232 7351 consensus_queue.cc:1035] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.906693 7351 consensus_queue.cc:1035] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:12.924373 7359 mvcc.cc:205] Tried to move back new op lower bound from 7316938067571036160 to 7316938067080167424. Current Snapshot: MvccSnapshot[applied={T|T < 7316938067571036160}]
I20260810 11:37:12.925489 7358 mvcc.cc:205] Tried to move back new op lower bound from 7316938067571036160 to 7316938067080167424. Current Snapshot: MvccSnapshot[applied={T|T < 7316938067571036160}]
W20260810 11:37:12.951310 7358 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 11:37:14.577898 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:14.578363 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35124:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:14.585330 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Sending DeleteTablet for 3 replicas of tablet 2f6b5984965249eeb75c0dea6c743f85
I20260810 11:37:14.586318 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 11:37:14.586308 6785 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53432
I20260810 11:37:14.587162 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 2f6b5984965249eeb75c0dea6c743f85 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47742
I20260810 11:37:14.587466 7434 tablet_replica.cc:330] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: stopping tablet replica
I20260810 11:37:14.587759 7434 raft_consensus.cc:2229] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:14.588223 7434 raft_consensus.cc:2258] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.592654 7434 ts_tablet_manager.cc:1905] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:14.599660 7436 tablet_replica.cc:330] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: stopping tablet replica
I20260810 11:37:14.599931 7436 raft_consensus.cc:2229] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:14.600342 7436 raft_consensus.cc:2258] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.601462 7436 ts_tablet_manager.cc:1905] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:14.601751 7437 tablet_replica.cc:330] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: stopping tablet replica
I20260810 11:37:14.602003 7437 raft_consensus.cc:2229] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:14.602699 7437 raft_consensus.cc:2258] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.603865 7437 ts_tablet_manager.cc:1905] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:14.627990 7434 ts_tablet_manager.cc:1918] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:14.628165 7434 log.cc:1196] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/2f6b5984965249eeb75c0dea6c743f85
I20260810 11:37:14.628619 7434 ts_tablet_manager.cc:1939] T 2f6b5984965249eeb75c0dea6c743f85 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting consensus metadata
I20260810 11:37:14.629715 7437 ts_tablet_manager.cc:1918] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:14.629925 7437 log.cc:1196] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/2f6b5984965249eeb75c0dea6c743f85
I20260810 11:37:14.630725 7437 ts_tablet_manager.cc:1939] T 2f6b5984965249eeb75c0dea6c743f85 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting consensus metadata
Aug 10 11:37:14 dist-test-slave-drl0 krb5kdc[6523](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 11:37:14 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:14.683658 7447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:14.658918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.3.240.254:36819)
0810 11:37:14.660235 (+ 1317us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:14.660240 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:14.660265 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:37:14.660327 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:14.660331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:14.660390 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:14.660483 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:14.662540 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:14.663697 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:14.665851 (+ 2154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:14.666355 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:14.666481 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:14.675912 (+ 9431us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:14.675919 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:14.675922 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:14.675962 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:14.679428 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:14.680361 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:14.680366 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:14.680368 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:14.680593 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:14.681236 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:14.681240 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:14.681310 (+ 70us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:14.682007 (+ 697us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:14.682321 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:14.682781 (+ 460us) server_negotiation.cc:299] Negotiation successful
0810 11:37:14.683112 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1167,"thread_start_us":945,"threads_started":1}
I20260810 11:37:14.688622 6584 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:14.689381 6584 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:14.698057 6785 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (DEFAULT_TABLE table=test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.698369 7189 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (DEFAULT_TABLE table=test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.698695 6785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.699379 6986 tablet_service.cc:1467] Processing CreateTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (DEFAULT_TABLE table=test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.699812 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.706604 7450 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Bootstrap starting.
I20260810 11:37:14.706434 7449 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Bootstrap starting.
I20260810 11:37:14.710088 7450 tablet_bootstrap.cc:654] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.710093 7449 tablet_bootstrap.cc:654] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.713161 7450 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: No bootstrap required, opened a new log
I20260810 11:37:14.713383 7450 ts_tablet_manager.cc:1397] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260810 11:37:14.713678 7449 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: No bootstrap required, opened a new log
I20260810 11:37:14.713943 7449 ts_tablet_manager.cc:1397] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 11:37:14.714619 7450 raft_consensus.cc:348] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.714855 7450 raft_consensus.cc:374] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.715042 7450 raft_consensus.cc:729] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f43ab78be084f8caaaea70d64d5ad63, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.715114 7449 raft_consensus.cc:348] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.715408 7449 raft_consensus.cc:374] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.715509 7449 raft_consensus.cc:729] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.715538 7450 consensus_queue.cc:260] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.715871 7449 consensus_queue.cc:260] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.716691 7449 ts_tablet_manager.cc:1428] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 11:37:14.716801 7450 ts_tablet_manager.cc:1428] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Time spent starting tablet: real 0.003s user 0.003s sys 0.001s
I20260810 11:37:14.719285 7436 ts_tablet_manager.cc:1918] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:14.719477 7436 log.cc:1196] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/2f6b5984965249eeb75c0dea6c743f85
I20260810 11:37:14.720031 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.720031 6564 catalog_manager.cc:4841] TS 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833): tablet 2f6b5984965249eeb75c0dea6c743f85 (table test-table [id=4e71ec56d818457fb0aee5542012a0a4]) successfully deleted
I20260810 11:37:14.720273 7436 ts_tablet_manager.cc:1939] T 2f6b5984965249eeb75c0dea6c743f85 P 02e57c0e7e424b8f9290163ce002facb: Deleting consensus metadata
I20260810 11:37:14.721771 6565 catalog_manager.cc:4841] TS 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641): tablet 2f6b5984965249eeb75c0dea6c743f85 (table test-table [id=4e71ec56d818457fb0aee5542012a0a4]) successfully deleted
I20260810 11:37:14.721859 6564 catalog_manager.cc:4841] TS 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743): tablet 2f6b5984965249eeb75c0dea6c743f85 (table test-table [id=4e71ec56d818457fb0aee5542012a0a4]) successfully deleted
I20260810 11:37:14.727412 7460 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Bootstrap starting.
I20260810 11:37:14.730345 7460 tablet_bootstrap.cc:654] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.733629 7460 tablet_bootstrap.cc:492] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: No bootstrap required, opened a new log
I20260810 11:37:14.733839 7460 ts_tablet_manager.cc:1397] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260810 11:37:14.735265 7460 raft_consensus.cc:348] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.735657 7460 raft_consensus.cc:374] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.735798 7460 raft_consensus.cc:729] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.736131 7460 consensus_queue.cc:260] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.737490 7460 ts_tablet_manager.cc:1428] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:14.785696 7462 raft_consensus.cc:482] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:14.786158 7462 raft_consensus.cc:504] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.787245 7462 leader_election.cc:288] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833), 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743)
I20260810 11:37:14.788024 6825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdc074a4a1b42ec9b0b2d0beb16ad39" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" is_pre_election: true
I20260810 11:37:14.788071 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdc074a4a1b42ec9b0b2d0beb16ad39" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63" is_pre_election: true
I20260810 11:37:14.788357 6825 raft_consensus.cc:2454] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 0.
I20260810 11:37:14.788357 7024 raft_consensus.cc:2454] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 0.
I20260810 11:37:14.788916 7122 leader_election.cc:302] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 1f43ab78be084f8caaaea70d64d5ad63; no voters:
I20260810 11:37:14.789228 7462 raft_consensus.cc:2781] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:14.789428 7462 raft_consensus.cc:482] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:14.789562 7462 raft_consensus.cc:3037] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:14.793185 7462 raft_consensus.cc:504] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.794027 7462 leader_election.cc:288] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 election: Requested vote from peers 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833), 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743)
I20260810 11:37:14.794687 6825 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdc074a4a1b42ec9b0b2d0beb16ad39" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab"
I20260810 11:37:14.794853 7024 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8fdc074a4a1b42ec9b0b2d0beb16ad39" candidate_uuid: "02e57c0e7e424b8f9290163ce002facb" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63"
I20260810 11:37:14.795125 6825 raft_consensus.cc:3037] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:14.795158 7024 raft_consensus.cc:3037] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:14.797821 7024 raft_consensus.cc:2454] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 1.
I20260810 11:37:14.798185 6825 raft_consensus.cc:2454] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02e57c0e7e424b8f9290163ce002facb in term 1.
I20260810 11:37:14.798324 7122 leader_election.cc:302] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 1f43ab78be084f8caaaea70d64d5ad63; no voters:
I20260810 11:37:14.798702 7462 raft_consensus.cc:2781] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:14.799378 7462 raft_consensus.cc:686] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 1 LEADER]: Becoming Leader. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Running, Role: LEADER
I20260810 11:37:14.799803 7462 consensus_queue.cc:237] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.803400 6584 catalog_manager.cc:5495] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb reported cstate change: term changed from 0 to 1, leader changed from <none> to 02e57c0e7e424b8f9290163ce002facb (127.3.240.195). New cstate: current_term: 1 leader_uuid: "02e57c0e7e424b8f9290163ce002facb" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } health_report { overall_health: HEALTHY } } }
I20260810 11:37:14.836601 7468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:14.825893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40158 (local address 127.3.240.195:40641)
0810 11:37:14.827487 (+ 1594us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:14.827492 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:14.827534 (+ 42us) server_negotiation.cc:407] Connection header received
0810 11:37:14.827615 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:14.827619 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:14.827693 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:14.827830 (+ 137us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:14.828744 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:14.831221 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:14.833601 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:14.834354 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:14.834520 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:14.835352 (+ 832us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:14.835480 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:14.835593 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:14.835760 (+ 167us) server_negotiation.cc:299] Negotiation successful
0810 11:37:14.835858 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1364,"thread_start_us":1058,"threads_started":1}
I20260810 11:37:14.850435 7024 raft_consensus.cc:1261] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Refusing update from remote peer 02e57c0e7e424b8f9290163ce002facb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:14.850783 6825 raft_consensus.cc:1261] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Refusing update from remote peer 02e57c0e7e424b8f9290163ce002facb: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:14.851294 7463 consensus_queue.cc:1035] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:14.851818 7462 consensus_queue.cc:1035] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [LEADER]: Connected to new peer: Peer: permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:14.911736 6582 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:14.912073 6582 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:14.916049 6582 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Sending DeleteTablet for 3 replicas of tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39
I20260810 11:37:14.916962 6785 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53432
I20260810 11:37:14.917186 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 11:37:14.917349 7437 tablet_replica.cc:330] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: stopping tablet replica
I20260810 11:37:14.917502 7434 tablet_replica.cc:330] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: stopping tablet replica
I20260810 11:37:14.917459 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:14 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47742
I20260810 11:37:14.917636 7437 raft_consensus.cc:2229] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:14.917762 7434 raft_consensus.cc:2229] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:14.917899 7437 raft_consensus.cc:2258] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.917900 7436 tablet_replica.cc:330] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: stopping tablet replica
I20260810 11:37:14.917994 7434 raft_consensus.cc:2258] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.918252 7436 raft_consensus.cc:2229] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:14.918874 7436 raft_consensus.cc:2258] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:14.919319 7437 ts_tablet_manager.cc:1905] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:14.919562 7434 ts_tablet_manager.cc:1905] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:14.918713 6582 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260810 11:37:14.920130 7436 ts_tablet_manager.cc:1905] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260810 11:37:14.920408 6582 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:14.928326 7434 ts_tablet_manager.cc:1918] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:14.929672 6785 tablet_service.cc:1467] Processing CreateTablet for tablet c64d1bf3638f4424ac80011bff37865f (DEFAULT_TABLE table=test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.929768 6986 tablet_service.cc:1467] Processing CreateTablet for tablet c64d1bf3638f4424ac80011bff37865f (DEFAULT_TABLE table=test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.930171 6785 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64d1bf3638f4424ac80011bff37865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.930259 6986 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64d1bf3638f4424ac80011bff37865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.931420 7436 ts_tablet_manager.cc:1918] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:14.931816 7436 log.cc:1196] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/8fdc074a4a1b42ec9b0b2d0beb16ad39
I20260810 11:37:14.932793 7436 ts_tablet_manager.cc:1939] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 02e57c0e7e424b8f9290163ce002facb: Deleting consensus metadata
I20260810 11:37:14.928619 7434 log.cc:1196] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/8fdc074a4a1b42ec9b0b2d0beb16ad39
I20260810 11:37:14.934870 7434 ts_tablet_manager.cc:1939] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 1f43ab78be084f8caaaea70d64d5ad63: Deleting consensus metadata
I20260810 11:37:14.935179 6565 catalog_manager.cc:4841] TS 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641): tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (table test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]) successfully deleted
I20260810 11:37:14.936595 7189 tablet_service.cc:1467] Processing CreateTablet for tablet c64d1bf3638f4424ac80011bff37865f (DEFAULT_TABLE table=test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:14.937536 7437 ts_tablet_manager.cc:1918] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:37:14.937919 7437 log.cc:1196] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/8fdc074a4a1b42ec9b0b2d0beb16ad39
I20260810 11:37:14.938664 7189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c64d1bf3638f4424ac80011bff37865f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:14.939000 6564 catalog_manager.cc:4841] TS 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743): tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (table test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]) successfully deleted
I20260810 11:37:14.940764 7450 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Bootstrap starting.
I20260810 11:37:14.942735 7437 ts_tablet_manager.cc:1939] T 8fdc074a4a1b42ec9b0b2d0beb16ad39 P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting consensus metadata
I20260810 11:37:14.944231 7450 tablet_bootstrap.cc:654] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.944236 6564 catalog_manager.cc:4841] TS 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833): tablet 8fdc074a4a1b42ec9b0b2d0beb16ad39 (table test-table [id=9da3b2eaa4484a909be87f0ebe533fb2]) successfully deleted
I20260810 11:37:14.947479 7449 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Bootstrap starting.
I20260810 11:37:14.950006 7450 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: No bootstrap required, opened a new log
I20260810 11:37:14.950312 7450 ts_tablet_manager.cc:1397] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 11:37:14.950452 7449 tablet_bootstrap.cc:654] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.951210 7460 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Bootstrap starting.
I20260810 11:37:14.951591 7450 raft_consensus.cc:348] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.952178 7450 raft_consensus.cc:374] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.952392 7450 raft_consensus.cc:729] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f43ab78be084f8caaaea70d64d5ad63, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.952996 7450 consensus_queue.cc:260] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.953950 7450 ts_tablet_manager.cc:1428] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Time spent starting tablet: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:14.955029 7460 tablet_bootstrap.cc:654] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:14.957624 7449 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: No bootstrap required, opened a new log
I20260810 11:37:14.958019 7449 ts_tablet_manager.cc:1397] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260810 11:37:14.959705 7460 tablet_bootstrap.cc:492] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: No bootstrap required, opened a new log
I20260810 11:37:14.959659 7449 raft_consensus.cc:348] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.960034 7460 ts_tablet_manager.cc:1397] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.004s
I20260810 11:37:14.960116 7449 raft_consensus.cc:374] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.960335 7449 raft_consensus.cc:729] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.960723 7449 consensus_queue.cc:260] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.961247 7460 raft_consensus.cc:348] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.961544 7449 ts_tablet_manager.cc:1428] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260810 11:37:14.961637 7460 raft_consensus.cc:374] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:14.961774 7460 raft_consensus.cc:729] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02e57c0e7e424b8f9290163ce002facb, State: Initialized, Role: FOLLOWER
I20260810 11:37:14.962386 7460 consensus_queue.cc:260] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:14.962921 7460 ts_tablet_manager.cc:1428] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260810 11:37:15.135063 7414 raft_consensus.cc:482] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:15.135340 7414 raft_consensus.cc:504] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:15.136446 7414 leader_election.cc:288] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:15.137149 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64d1bf3638f4424ac80011bff37865f" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63" is_pre_election: true
I20260810 11:37:15.137182 7216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64d1bf3638f4424ac80011bff37865f" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb" is_pre_election: true
I20260810 11:37:15.137581 7022 raft_consensus.cc:2454] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:15.137691 7216 raft_consensus.cc:2454] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 0.
I20260810 11:37:15.138227 6715 leader_election.cc:302] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1f43ab78be084f8caaaea70d64d5ad63, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:15.138649 7414 raft_consensus.cc:2781] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:15.138880 7414 raft_consensus.cc:482] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:15.139065 7414 raft_consensus.cc:3037] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:15.142356 7414 raft_consensus.cc:504] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:15.143131 7414 leader_election.cc:288] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Requested vote from peers 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743), 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641)
I20260810 11:37:15.143862 7216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64d1bf3638f4424ac80011bff37865f" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "02e57c0e7e424b8f9290163ce002facb"
I20260810 11:37:15.143913 7022 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c64d1bf3638f4424ac80011bff37865f" candidate_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1f43ab78be084f8caaaea70d64d5ad63"
I20260810 11:37:15.144207 7216 raft_consensus.cc:3037] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:15.144232 7022 raft_consensus.cc:3037] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:15.149058 7022 raft_consensus.cc:2454] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:15.149109 7216 raft_consensus.cc:2454] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 35ae26b2a6f748ef89a4fec8f7f7d5ab in term 1.
I20260810 11:37:15.149612 6716 leader_election.cc:302] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 02e57c0e7e424b8f9290163ce002facb, 35ae26b2a6f748ef89a4fec8f7f7d5ab; no voters:
I20260810 11:37:15.150091 7414 raft_consensus.cc:2781] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:15.150307 7414 raft_consensus.cc:686] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 LEADER]: Becoming Leader. State: Replica: 35ae26b2a6f748ef89a4fec8f7f7d5ab, State: Running, Role: LEADER
I20260810 11:37:15.150717 7414 consensus_queue.cc:237] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } }
I20260810 11:37:15.154572 6582 catalog_manager.cc:5495] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab reported cstate change: term changed from 0 to 1, leader changed from <none> to 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193). New cstate: current_term: 1 leader_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "35ae26b2a6f748ef89a4fec8f7f7d5ab" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 } health_report { overall_health: UNKNOWN } } }
I20260810 11:37:15.219949 7479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:15.210522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54090 (local address 127.3.240.193:43833)
0810 11:37:15.211895 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:15.211899 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:15.212006 (+ 107us) server_negotiation.cc:407] Connection header received
0810 11:37:15.212076 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:15.212079 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:15.212147 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:15.212294 (+ 147us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:15.213116 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:15.214663 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:15.216807 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:15.217694 (+ 887us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:15.217861 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:15.218583 (+ 722us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:15.218783 (+ 200us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:15.218894 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:15.219078 (+ 184us) server_negotiation.cc:299] Negotiation successful
0810 11:37:15.219160 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1166,"thread_start_us":909,"threads_started":1}
I20260810 11:37:15.231421 7216 raft_consensus.cc:1261] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:15.231561 7022 raft_consensus.cc:1261] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Refusing update from remote peer 35ae26b2a6f748ef89a4fec8f7f7d5ab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:15.232198 7414 consensus_queue.cc:1035] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "02e57c0e7e424b8f9290163ce002facb" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40641 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:15.232625 7406 consensus_queue.cc:1035] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [LEADER]: Connected to new peer: Peer: permanent_uuid: "1f43ab78be084f8caaaea70d64d5ad63" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 11:37:15.257257 7404 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260810 11:37:16.849412 6584 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:16.849743 6584 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55272:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:16.855444 6584 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 842ddf17916d46dabcba4dca17922fad: Sending DeleteTablet for 3 replicas of tablet c64d1bf3638f4424ac80011bff37865f
I20260810 11:37:16.856313 6785 tablet_service.cc:1514] Processing DeleteTablet for tablet c64d1bf3638f4424ac80011bff37865f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53432
I20260810 11:37:16.856499 6986 tablet_service.cc:1514] Processing DeleteTablet for tablet c64d1bf3638f4424ac80011bff37865f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:34114
I20260810 11:37:16.856577 7189 tablet_service.cc:1514] Processing DeleteTablet for tablet c64d1bf3638f4424ac80011bff37865f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:16 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47742
I20260810 11:37:16.857214 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6687
I20260810 11:37:16.867561 7518 tablet_replica.cc:330] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: stopping tablet replica
I20260810 11:37:16.867869 7518 raft_consensus.cc:2229] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:16.868189 7518 raft_consensus.cc:2258] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:16.868476 7516 tablet_replica.cc:330] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: stopping tablet replica
I20260810 11:37:16.868955 7517 tablet_replica.cc:330] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: stopping tablet replica
I20260810 11:37:16.869165 7517 raft_consensus.cc:2229] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:16.869447 7517 raft_consensus.cc:2258] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:16.869704 7516 raft_consensus.cc:2229] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:16.869783 7518 ts_tablet_manager.cc:1905] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:16.870157 7516 raft_consensus.cc:2258] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:16.870889 7517 ts_tablet_manager.cc:1905] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:16.871275 7516 ts_tablet_manager.cc:1905] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:16.878199 7518 ts_tablet_manager.cc:1918] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:16.878443 7518 log.cc:1196] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/c64d1bf3638f4424ac80011bff37865f
I20260810 11:37:16.878464 7517 ts_tablet_manager.cc:1918] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:16.878727 7517 log.cc:1196] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/c64d1bf3638f4424ac80011bff37865f
I20260810 11:37:16.879091 7518 ts_tablet_manager.cc:1939] T c64d1bf3638f4424ac80011bff37865f P 02e57c0e7e424b8f9290163ce002facb: Deleting consensus metadata
I20260810 11:37:16.879377 7516 ts_tablet_manager.cc:1918] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260810 11:37:16.879426 7517 ts_tablet_manager.cc:1939] T c64d1bf3638f4424ac80011bff37865f P 1f43ab78be084f8caaaea70d64d5ad63: Deleting consensus metadata
I20260810 11:37:16.879534 7516 log.cc:1196] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/c64d1bf3638f4424ac80011bff37865f
I20260810 11:37:16.880090 7516 ts_tablet_manager.cc:1939] T c64d1bf3638f4424ac80011bff37865f P 35ae26b2a6f748ef89a4fec8f7f7d5ab: Deleting consensus metadata
I20260810 11:37:16.880497 6565 catalog_manager.cc:4841] TS 02e57c0e7e424b8f9290163ce002facb (127.3.240.195:40641): tablet c64d1bf3638f4424ac80011bff37865f (table test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]) successfully deleted
I20260810 11:37:16.881274 6564 catalog_manager.cc:4841] TS 1f43ab78be084f8caaaea70d64d5ad63 (127.3.240.194:38743): tablet c64d1bf3638f4424ac80011bff37865f (table test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]) successfully deleted
I20260810 11:37:16.881526 6564 catalog_manager.cc:4841] TS 35ae26b2a6f748ef89a4fec8f7f7d5ab (127.3.240.193:43833): tablet c64d1bf3638f4424ac80011bff37865f (table test-table [id=1dabfac4b1f84ef9bd05b4d80c0edf76]) successfully deleted
Aug 10 11:37:16 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:16.889781 7520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:16.859446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54096 (local address 127.3.240.193:43833)
0810 11:37:16.869082 (+ 9636us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:16.869088 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:16.869118 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:37:16.869186 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:16.869193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:16.869278 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:16.869422 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:16.871479 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.872865 (+ 1386us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.875242 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.876244 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.876404 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:16.882739 (+ 6335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:16.882755 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:16.882772 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:16.882804 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:16.886374 (+ 3570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:16.887305 (+ 931us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:16.887312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:16.887315 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:16.887480 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:16.887991 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:16.887995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:16.887998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:16.888408 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:16.888717 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:16.888986 (+ 269us) server_negotiation.cc:299] Negotiation successful
0810 11:37:16.889163 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9487,"thread_start_us":8973,"threads_started":1}
W20260810 11:37:16.891543 6879 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54096
W20260810 11:37:16.892714 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6687: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:16.893054 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 6687
I20260810 11:37:16.936476 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6892
Aug 10 11:37:16 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:16.958503 7521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:16.937703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48944 (local address 127.3.240.194:38743)
0810 11:37:16.938363 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:16.938370 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:16.938405 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:37:16.938472 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:16.938478 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:16.938564 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:16.938689 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:16.940376 (+ 1687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.941122 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.942540 (+ 1418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.943339 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.943445 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:16.949233 (+ 5788us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:16.949262 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:16.949288 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:16.949334 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:16.955081 (+ 5747us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:16.955919 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:16.955927 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:16.955929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:16.956106 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:16.956560 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:16.956564 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:16.956566 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:16.956975 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:16.957218 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:16.957812 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 11:37:16.958046 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":371,"threads_started":1}
W20260810 11:37:16.959605 7082 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:48944
W20260810 11:37:16.960482 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6892: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:16.960634 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 6892
I20260810 11:37:16.989823 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7092
Aug 10 11:37:16 dist-test-slave-drl0 krb5kdc[6523](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361834, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:17.008818 7522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:16.991027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40166 (local address 127.3.240.195:40641)
0810 11:37:16.991609 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:16.991613 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:16.991658 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:37:16.991733 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:16.991736 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:16.991807 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:16.991947 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:16.993563 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.994325 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.995615 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:16.996009 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:16.996153 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:17.002274 (+ 6121us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:17.002287 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:17.002306 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:17.002340 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:17.005786 (+ 3446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:17.006596 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:17.006604 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:17.006606 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:17.006739 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:17.007270 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:17.007274 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:17.007276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:17.007635 (+ 359us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:17.007989 (+ 354us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:17.008244 (+ 255us) server_negotiation.cc:299] Negotiation successful
0810 11:37:17.008464 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":468,"thread_start_us":326,"threads_started":1}
W20260810 11:37:17.009541 7276 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40166
W20260810 11:37:17.010248 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7092: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:17.010470 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 7092
I20260810 11:37:17.041056 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 6541
I20260810 11:37:17.042367 6643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:17.174643 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 6541
2026-08-10T11:37:17Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9739 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:17 dist-test-slave-drl0 krb5kdc[7529](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:17 dist-test-slave-drl0 krb5kdc[7529](info): set up 2 sockets
Aug 10 11:37:17 dist-test-slave-drl0 krb5kdc[7529](info): commencing operation
krb5kdc: starting...
W20260810 11:37:18.534029 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.003s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:18 dist-test-slave-drl0 krb5kdc[7529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:18.780395 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44641
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44641
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:19.156454 7548 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:19.156929 7548 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:19.157097 7548 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:19.164691 7548 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:19.164837 7548 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:19.164942 7548 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:19.165038 7548 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:19.165123 7548 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:19.175302 7548 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44641
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44641
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:19.176853 7548 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:19.178347 7548 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:19.193276 7558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:19.193181 7565 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:19.193245 7559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:19.194136 7548 server_base.cc:1034] running on GCE node
I20260810 11:37:19.195081 7548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:19.196705 7548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:19.197971 7548 hybrid_clock.cc:648] HybridClock initialized: now 1786361839197902 us; error 49 us; skew 500 ppm
I20260810 11:37:19.198362 7548 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:19 dist-test-slave-drl0 krb5kdc[7529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361839, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:19.201975 7548 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:19.203663 7548 webserver.cc:460] Webserver started at http://127.3.240.254:33763/ using document root <none> and password file <none>
I20260810 11:37:19.204373 7548 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:19.204522 7548 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:19.204861 7548 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:19.208452 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "298bbd5860e4474d88849cb655a73839"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "128bc4861e338b297a497d01da14574f"
server_key_iv: "91be25f0141719c99b89bd255c57ccc1"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.209333 7548 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "298bbd5860e4474d88849cb655a73839"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "128bc4861e338b297a497d01da14574f"
server_key_iv: "91be25f0141719c99b89bd255c57ccc1"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.215052 7548 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:37:19.218546 7575 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:19.220022 7548 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:19.220366 7548 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "298bbd5860e4474d88849cb655a73839"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "128bc4861e338b297a497d01da14574f"
server_key_iv: "91be25f0141719c99b89bd255c57ccc1"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.220629 7548 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:19.241956 7548 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:19.246495 7548 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:19.246836 7548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:19.270133 7548 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44641
I20260810 11:37:19.270169 7648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44641 every 8 connection(s)
I20260810 11:37:19.271651 7548 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:19.275731 7649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:19.278353 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 7548
I20260810 11:37:19.278601 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:19.279327 4035 external_mini_cluster.cc:1428] Setting key 38a1eeac3419a1035063572bf03e7d65
I20260810 11:37:19.289036 7649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839: Bootstrap starting.
I20260810 11:37:19.293171 7649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:19.294729 7649 log.cc:824] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:37:19 dist-test-slave-drl0 krb5kdc[7529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361838, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:19.298177 7649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839: No bootstrap required, opened a new log
I20260810 11:37:19.306649 7654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:19.281366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35724 (local address 127.3.240.254:44641)
0810 11:37:19.282109 (+ 743us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:19.282121 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:19.282162 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:37:19.284026 (+ 1864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:19.284048 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:19.284558 (+ 510us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:19.284900 (+ 342us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:19.287355 (+ 2455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.288640 (+ 1285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.290528 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.291322 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.291482 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:19.297286 (+ 5804us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:19.297297 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:19.297317 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:19.297366 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:19.301613 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:19.302337 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:19.302345 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:19.302354 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:19.302568 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:19.303021 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:19.303026 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:19.303029 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:19.303729 (+ 700us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:19.304030 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:19.304707 (+ 677us) server_negotiation.cc:299] Negotiation successful
0810 11:37:19.305177 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":341,"threads_started":1}
I20260810 11:37:19.309947 7649 raft_consensus.cc:348] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } }
I20260810 11:37:19.310452 7649 raft_consensus.cc:374] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:19.310577 7649 raft_consensus.cc:729] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 298bbd5860e4474d88849cb655a73839, State: Initialized, Role: FOLLOWER
I20260810 11:37:19.311275 7649 consensus_queue.cc:260] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } }
I20260810 11:37:19.311550 7649 raft_consensus.cc:388] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:19.311722 7649 raft_consensus.cc:482] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:19.311928 7649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:19.314462 7649 raft_consensus.cc:504] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } }
I20260810 11:37:19.315384 7649 leader_election.cc:302] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 298bbd5860e4474d88849cb655a73839; no voters:
I20260810 11:37:19.316246 7649 leader_election.cc:288] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:19.316392 7657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:19.317735 7657 raft_consensus.cc:686] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [term 1 LEADER]: Becoming Leader. State: Replica: 298bbd5860e4474d88849cb655a73839, State: Running, Role: LEADER
I20260810 11:37:19.318459 7649 sys_catalog.cc:564] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:19.318341 7657 consensus_queue.cc:237] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } }
I20260810 11:37:19.320984 7662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 298bbd5860e4474d88849cb655a73839. Latest consensus state: current_term: 1 leader_uuid: "298bbd5860e4474d88849cb655a73839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } } }
I20260810 11:37:19.321933 7662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:19.321846 7660 sys_catalog.cc:455] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "298bbd5860e4474d88849cb655a73839" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "298bbd5860e4474d88849cb655a73839" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44641 } } }
I20260810 11:37:19.322504 7660 sys_catalog.cc:458] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:19.324517 7667 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:19.330426 7667 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:19.338819 7667 catalog_manager.cc:1347] Generated new cluster ID: 09659cbcce9540129e38d6a9cc1c7952
I20260810 11:37:19.339006 7667 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:19.354306 7667 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:19.355767 7667 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:19.364365 7667 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 298bbd5860e4474d88849cb655a73839: Generated new TSK 0
I20260810 11:37:19.365176 7667 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:19.429306 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:19.803224 7690 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:19.803550 7690 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:19.803718 7690 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:19.812151 7690 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:19.812316 7690 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:19.812494 7690 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:19.823686 7690 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:19.825613 7690 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:19.827406 7690 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:19.844121 7699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:19.844070 7698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:19.844172 7702 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:19.845211 7690 server_base.cc:1034] running on GCE node
I20260810 11:37:19.845798 7690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:19.846834 7690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:19.848187 7690 hybrid_clock.cc:648] HybridClock initialized: now 1786361839848106 us; error 81 us; skew 500 ppm
I20260810 11:37:19.848573 7690 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:19 dist-test-slave-drl0 krb5kdc[7529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361839, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:19.856076 7690 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:19.857712 7690 webserver.cc:460] Webserver started at http://127.3.240.193:39535/ using document root <none> and password file <none>
I20260810 11:37:19.858594 7690 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:19.858760 7690 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:19.859156 7690 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:19.862157 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "a6d4968496b247e18db35c1e0c2df983"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "429fb8dcc742bef1d05f41ac983c3281"
server_key_iv: "1db7c0bb683d7bb2ae4fe9b4be1cabed"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.863083 7690 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "a6d4968496b247e18db35c1e0c2df983"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "429fb8dcc742bef1d05f41ac983c3281"
server_key_iv: "1db7c0bb683d7bb2ae4fe9b4be1cabed"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.868920 7690 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 11:37:19.872833 7710 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:19.874569 7690 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:37:19.874799 7690 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "a6d4968496b247e18db35c1e0c2df983"
format_stamp: "Formatted at 2026-08-10 11:37:19 on dist-test-slave-drl0"
server_key: "429fb8dcc742bef1d05f41ac983c3281"
server_key_iv: "1db7c0bb683d7bb2ae4fe9b4be1cabed"
server_key_version: "encryptionkey@0"
I20260810 11:37:19.875083 7690 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:19.891992 7690 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:19.897297 7690 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:19.897625 7690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:19.898795 7690 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:19.900238 7690 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:19.900347 7690 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:19.900650 7690 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:19.900767 7690 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:19.944324 7690 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44293
I20260810 11:37:19.944461 7875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44293 every 8 connection(s)
I20260810 11:37:19.945991 7690 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:19.955241 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 7690
I20260810 11:37:19.955488 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:19.956132 4035 external_mini_cluster.cc:1428] Setting key 68b592f6ed6894dbfa756b86b21618ab
Aug 10 11:37:19 dist-test-slave-drl0 krb5kdc[7529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361839, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:19.968438 7878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:19.949110 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57705 (local address 127.3.240.254:44641)
0810 11:37:19.949645 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:19.949650 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:19.950679 (+ 1029us) server_negotiation.cc:407] Connection header received
0810 11:37:19.952959 (+ 2280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:19.952965 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:19.953064 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:19.953208 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:19.954843 (+ 1635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.955850 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.958088 (+ 2238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.958598 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.958738 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:19.962552 (+ 3814us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:19.962557 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:19.962561 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:19.962592 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:19.965270 (+ 2678us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:19.966038 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:19.966051 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:19.966053 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:19.966166 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:19.966603 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:19.966608 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:19.966612 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:19.966906 (+ 294us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:19.967107 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:19.967848 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 11:37:19.968068 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":290,"threads_started":1}
I20260810 11:37:19.969453 7880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:19.949640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44641 (local address 127.3.240.193:57705)
0810 11:37:19.950431 (+ 791us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:19.950478 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:19.952640 (+ 2162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:19.953458 (+ 818us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:19.953466 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:19.954018 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:19.954633 (+ 615us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:19.954651 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.956029 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.956034 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:19.957866 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:19.957884 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:19.959097 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:19.959113 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:19.959352 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:19.960081 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:19.960105 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:19.962319 (+ 2214us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:19.965445 (+ 3126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:19.965455 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:19.965471 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:19.965901 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:19.966282 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:19.966287 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:19.966291 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:19.966484 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:19.967082 (+ 598us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:19.967090 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:19.967431 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 11:37:19.967734 (+ 303us) client_negotiation.cc:240] Negotiation successful
0810 11:37:19.968283 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":309,"threads_started":1}
I20260810 11:37:19.971254 7876 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44641
I20260810 11:37:19.971612 7876 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:19.972390 7876 heartbeater.cc:511] Master 127.3.240.254:44641 requested a full tablet report, sending...
I20260810 11:37:19.974627 7598 ts_manager.cc:194] Registered new tserver with Master: a6d4968496b247e18db35c1e0c2df983 (127.3.240.193:44293)
I20260810 11:37:19.976193 7598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:57705
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:20.015021 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:20.390918 7885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:20.391281 7885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:20.391386 7885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:20.399595 7885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:20.399708 7885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:20.399824 7885 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:20.409825 7885 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:20.411113 7885 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:20.412750 7885 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:20.428175 7896 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:20.428258 7894 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:20.428799 7893 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:20.430099 7885 server_base.cc:1034] running on GCE node
I20260810 11:37:20.430974 7885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:20.432127 7885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:20.433394 7885 hybrid_clock.cc:648] HybridClock initialized: now 1786361840433340 us; error 50 us; skew 500 ppm
I20260810 11:37:20.433763 7885 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:20 dist-test-slave-drl0 krb5kdc[7529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361840, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:20.438019 7885 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:20.439797 7885 webserver.cc:460] Webserver started at http://127.3.240.194:42729/ using document root <none> and password file <none>
I20260810 11:37:20.440546 7885 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:20.440687 7885 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:20.441059 7885 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:20.444494 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "fca9bc0c9aaf4663829fd9ba4b3b0f0e"
format_stamp: "Formatted at 2026-08-10 11:37:20 on dist-test-slave-drl0"
server_key: "114bb1ea4f5c3c947c2f2946ef2101ff"
server_key_iv: "bab85ba2a27523ee6023b76fc5415997"
server_key_version: "encryptionkey@0"
I20260810 11:37:20.445348 7885 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "fca9bc0c9aaf4663829fd9ba4b3b0f0e"
format_stamp: "Formatted at 2026-08-10 11:37:20 on dist-test-slave-drl0"
server_key: "114bb1ea4f5c3c947c2f2946ef2101ff"
server_key_iv: "bab85ba2a27523ee6023b76fc5415997"
server_key_version: "encryptionkey@0"
I20260810 11:37:20.450901 7885 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 11:37:20.454844 7911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:20.456293 7885 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:20.456453 7885 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "fca9bc0c9aaf4663829fd9ba4b3b0f0e"
format_stamp: "Formatted at 2026-08-10 11:37:20 on dist-test-slave-drl0"
server_key: "114bb1ea4f5c3c947c2f2946ef2101ff"
server_key_iv: "bab85ba2a27523ee6023b76fc5415997"
server_key_version: "encryptionkey@0"
I20260810 11:37:20.456715 7885 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:20.491559 7885 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:20.496551 7885 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:20.496881 7885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:20.497958 7885 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:20.499574 7885 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:20.499652 7885 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:20.499783 7885 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:20.499845 7885 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:20.541404 7885 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38349
I20260810 11:37:20.541491 8061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38349 every 8 connection(s)
I20260810 11:37:20.542740 7885 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:20.551578 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 7885
I20260810 11:37:20.551823 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:20.552421 4035 external_mini_cluster.cc:1428] Setting key 3b619bc0657616be5605036cc50b2bd5
Aug 10 11:37:20 dist-test-slave-drl0 krb5kdc[7529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361840, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:20.568604 8065 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:20.545701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54907 (local address 127.3.240.254:44641)
0810 11:37:20.546403 (+ 702us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:20.546410 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:20.547269 (+ 859us) server_negotiation.cc:407] Connection header received
0810 11:37:20.549514 (+ 2245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:20.549519 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:20.549594 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:20.549704 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:20.552146 (+ 2442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:20.553234 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:20.555333 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:20.555838 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:20.556000 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:20.560305 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:20.560311 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:20.560316 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:20.560354 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:20.564504 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:20.565339 (+ 835us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:20.565347 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:20.565353 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:20.565530 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:20.566029 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:20.566037 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:20.566043 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:20.566460 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:20.566622 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:20.567805 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0810 11:37:20.568093 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":387,"threads_started":1}
I20260810 11:37:20.569713 8066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:20.546246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44641 (local address 127.3.240.194:54907)
0810 11:37:20.547105 (+ 859us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:20.547155 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:20.549203 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:20.550163 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:20.550177 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:20.550867 (+ 690us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:20.551851 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:20.551886 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:20.553469 (+ 1583us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:20.553475 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:20.555078 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:20.555121 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:20.556148 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:20.556157 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:20.556443 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:20.557318 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:20.557347 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:20.560018 (+ 2671us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:20.564701 (+ 4683us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:20.564708 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:20.564722 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:20.565183 (+ 461us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:20.565682 (+ 499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:20.565686 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:20.565689 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:20.565895 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:20.566618 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:20.566625 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:20.567307 (+ 682us) client_negotiation.cc:769] Sending connection context
0810 11:37:20.567634 (+ 327us) client_negotiation.cc:240] Negotiation successful
0810 11:37:20.568209 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":314,"threads_started":1}
I20260810 11:37:20.571827 8063 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44641
I20260810 11:37:20.572189 8063 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:20.573004 8063 heartbeater.cc:511] Master 127.3.240.254:44641 requested a full tablet report, sending...
I20260810 11:37:20.575287 7598 ts_manager.cc:194] Registered new tserver with Master: fca9bc0c9aaf4663829fd9ba4b3b0f0e (127.3.240.194:38349)
I20260810 11:37:20.576170 7598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:54907
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:20.609987 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:20.978571 8070 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:20.978914 8070 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:20.979111 8070 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260810 11:37:20.979441 7876 heartbeater.cc:503] Master 127.3.240.254:44641 was elected leader, sending a full tablet report...
W20260810 11:37:20.987514 8070 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:20.987709 8070 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:20.987906 8070 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:20.999184 8070 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:44641
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:21.000573 8070 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:21.001959 8070 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:21.020191 8076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:21.020218 8079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:21.020273 8081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:21.020826 8070 server_base.cc:1034] running on GCE node
I20260810 11:37:21.021543 8070 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:21.022517 8070 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:21.024122 8070 hybrid_clock.cc:648] HybridClock initialized: now 1786361841023828 us; error 53 us; skew 500 ppm
I20260810 11:37:21.024735 8070 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:21 dist-test-slave-drl0 krb5kdc[7529](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:21.028824 8070 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:21.030382 8070 webserver.cc:460] Webserver started at http://127.3.240.195:45267/ using document root <none> and password file <none>
I20260810 11:37:21.031177 8070 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:21.031324 8070 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:21.031644 8070 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:21.034583 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "90ae9db923eb4df792a339aea157f503"
format_stamp: "Formatted at 2026-08-10 11:37:21 on dist-test-slave-drl0"
server_key: "de2a6caf4324682ac737611a294132ce"
server_key_iv: "cdfd7c62d48bd4c52b9d99261feb646d"
server_key_version: "encryptionkey@0"
I20260810 11:37:21.035411 8070 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "90ae9db923eb4df792a339aea157f503"
format_stamp: "Formatted at 2026-08-10 11:37:21 on dist-test-slave-drl0"
server_key: "de2a6caf4324682ac737611a294132ce"
server_key_iv: "cdfd7c62d48bd4c52b9d99261feb646d"
server_key_version: "encryptionkey@0"
I20260810 11:37:21.041411 8070 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 11:37:21.046607 8088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:21.048121 8070 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 11:37:21.048324 8070 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "90ae9db923eb4df792a339aea157f503"
format_stamp: "Formatted at 2026-08-10 11:37:21 on dist-test-slave-drl0"
server_key: "de2a6caf4324682ac737611a294132ce"
server_key_iv: "cdfd7c62d48bd4c52b9d99261feb646d"
server_key_version: "encryptionkey@0"
I20260810 11:37:21.048554 8070 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:21.071748 8070 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:21.077322 8070 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:21.077646 8070 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:21.078835 8070 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:21.080368 8070 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:21.080479 8070 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:21.080623 8070 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:21.080695 8070 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:21.122008 8070 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:39567
I20260810 11:37:21.122054 8247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:39567 every 8 connection(s)
I20260810 11:37:21.123427 8070 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:21.126511 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8070
I20260810 11:37:21.126742 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:21.127353 4035 external_mini_cluster.cc:1428] Setting key f4004685690e4200ed1d4b30036b18e4
Aug 10 11:37:21 dist-test-slave-drl0 krb5kdc[7529](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361841, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:21.149106 8252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:21.128165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:36185 (local address 127.3.240.254:44641)
0810 11:37:21.128686 (+ 521us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:21.128692 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:21.129335 (+ 643us) server_negotiation.cc:407] Connection header received
0810 11:37:21.132225 (+ 2890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:21.132229 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:21.132313 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:21.132420 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:21.134583 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:21.135467 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:21.137206 (+ 1739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:21.137629 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:21.137736 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:21.141698 (+ 3962us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:21.141703 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:21.141706 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:21.141747 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:21.145225 (+ 3478us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:21.146183 (+ 958us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:21.146190 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:21.146193 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:21.146431 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:21.147052 (+ 621us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:21.147057 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:21.147060 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:21.147409 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:21.147557 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:21.148464 (+ 907us) server_negotiation.cc:299] Negotiation successful
0810 11:37:21.148743 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":279,"threads_started":1}
I20260810 11:37:21.149909 8251 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:21.127223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:44641 (local address 127.3.240.195:36185)
0810 11:37:21.128282 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:21.128333 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:21.131917 (+ 3584us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:21.132824 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:21.132835 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:21.133526 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:21.134336 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:21.134360 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:21.135674 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:21.135680 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:21.137026 (+ 1346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:21.137041 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:21.137867 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:21.137872 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:21.138094 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:21.138838 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:21.138862 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:21.141468 (+ 2606us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:21.145475 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:21.145482 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:21.145495 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:21.145948 (+ 453us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:21.146622 (+ 674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:21.146625 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:21.146628 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:21.146821 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:21.147578 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:21.147585 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:21.147963 (+ 378us) client_negotiation.cc:769] Sending connection context
0810 11:37:21.148238 (+ 275us) client_negotiation.cc:240] Negotiation successful
0810 11:37:21.148671 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":494,"threads_started":1}
I20260810 11:37:21.151671 8248 heartbeater.cc:347] Connected to a master server at 127.3.240.254:44641
I20260810 11:37:21.152041 8248 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:21.152827 8248 heartbeater.cc:511] Master 127.3.240.254:44641 requested a full tablet report, sending...
I20260810 11:37:21.154814 7598 ts_manager.cc:194] Registered new tserver with Master: 90ae9db923eb4df792a339aea157f503 (127.3.240.195:39567)
I20260810 11:37:21.155860 7598 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:36185
I20260810 11:37:21.156631 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 11:37:21.198504 8252 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:21.195683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35738 (local address 127.3.240.254:44641)
0810 11:37:21.195998 (+ 315us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:21.196008 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:21.196311 (+ 303us) server_negotiation.cc:407] Connection header received
0810 11:37:21.196516 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:21.196531 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:21.196611 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:21.196728 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:21.198183 (+ 1455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35738: BlockingRecv error: recv got EOF from 127.0.0.1:35738 (error 108)
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 11:37:21.202602 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7690
W20260810 11:37:21.206022 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7690: Not authorized: Client connection negotiation failed: client connection to 127.3.240.193:44293: server requires authentication, but client does not have Kerberos credentials available
I20260810 11:37:21.206238 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 7690
I20260810 11:37:21.239804 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7885
W20260810 11:37:21.243151 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7885: Not authorized: Client connection negotiation failed: client connection to 127.3.240.194:38349: server requires authentication, but client does not have Kerberos credentials available
I20260810 11:37:21.243333 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 7885
I20260810 11:37:21.266222 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8070
W20260810 11:37:21.270009 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8070: Not authorized: Client connection negotiation failed: client connection to 127.3.240.195:39567: server requires authentication, but client does not have Kerberos credentials available
I20260810 11:37:21.270169 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8070
I20260810 11:37:21.292733 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 7548
I20260810 11:37:21.294166 7640 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:21.403105 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 7548
2026-08-10T11:37:21Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4218 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:21 dist-test-slave-drl0 krb5kdc[8271](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:21 dist-test-slave-drl0 krb5kdc[8271](info): set up 2 sockets
Aug 10 11:37:21 dist-test-slave-drl0 krb5kdc[8271](info): commencing operation
krb5kdc: starting...
W20260810 11:37:22.762027 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.304s user 0.001s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:22 dist-test-slave-drl0 krb5kdc[8271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:23.019971 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41867
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:41867
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:23.387471 8288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:23.387857 8288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:23.388028 8288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:23.395975 8288 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:23.396136 8288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:23.396263 8288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:23.396381 8288 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:23.396473 8288 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:23.409588 8288 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:41867
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:23.411123 8288 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:23.412580 8288 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:23.425575 8294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:23.425474 8298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:23.426004 8295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:23.426815 8288 server_base.cc:1034] running on GCE node
I20260810 11:37:23.427735 8288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:23.429179 8288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:23.430435 8288 hybrid_clock.cc:648] HybridClock initialized: now 1786361843430398 us; error 36 us; skew 500 ppm
I20260810 11:37:23.430877 8288 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:23 dist-test-slave-drl0 krb5kdc[8271](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361843, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:23.434227 8288 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:23.437772 8288 webserver.cc:460] Webserver started at http://127.3.240.254:37157/ using document root <none> and password file <none>
I20260810 11:37:23.438510 8288 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:23.438644 8288 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:23.439069 8288 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:23.442159 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "431d20ac026f470e9a0d9d6ebb1842eb"
format_stamp: "Formatted at 2026-08-10 11:37:23 on dist-test-slave-drl0"
server_key: "41322d00a4f1a0e1c63b8c9f2da67b9d"
server_key_iv: "83bf5415523b26c633814ff920bc27af"
server_key_version: "encryptionkey@0"
I20260810 11:37:23.443019 8288 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "431d20ac026f470e9a0d9d6ebb1842eb"
format_stamp: "Formatted at 2026-08-10 11:37:23 on dist-test-slave-drl0"
server_key: "41322d00a4f1a0e1c63b8c9f2da67b9d"
server_key_iv: "83bf5415523b26c633814ff920bc27af"
server_key_version: "encryptionkey@0"
I20260810 11:37:23.448587 8288 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260810 11:37:23.452872 8309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:23.454473 8288 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:23.454680 8288 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "431d20ac026f470e9a0d9d6ebb1842eb"
format_stamp: "Formatted at 2026-08-10 11:37:23 on dist-test-slave-drl0"
server_key: "41322d00a4f1a0e1c63b8c9f2da67b9d"
server_key_iv: "83bf5415523b26c633814ff920bc27af"
server_key_version: "encryptionkey@0"
I20260810 11:37:23.454877 8288 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:23.476701 8288 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:23.482517 8288 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:23.482966 8288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:23.505398 8288 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:41867
I20260810 11:37:23.505402 8375 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:41867 every 8 connection(s)
I20260810 11:37:23.507179 8288 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:23.508718 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8288
I20260810 11:37:23.508991 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:23.510162 4035 external_mini_cluster.cc:1428] Setting key 6b18072a8edb8acbec11a6b5078c51b7
I20260810 11:37:23.513063 8378 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:23.526347 8378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb: Bootstrap starting.
Aug 10 11:37:23 dist-test-slave-drl0 krb5kdc[8271](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361842, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:23.529978 8378 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:23.531374 8378 log.cc:824] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:23.534199 8378 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb: No bootstrap required, opened a new log
I20260810 11:37:23.537660 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:23.512441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56036 (local address 127.3.240.254:41867)
0810 11:37:23.513530 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:23.513543 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:23.513588 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:37:23.515496 (+ 1908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:23.515520 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:23.515944 (+ 424us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:23.516371 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:23.518082 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:23.520014 (+ 1932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:23.521929 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:23.523012 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:23.523238 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:23.528695 (+ 5457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:23.528706 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:23.528729 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:23.528794 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:23.532876 (+ 4082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:23.533798 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:23.533808 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:23.533815 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:23.534002 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:23.534500 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:23.534515 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:23.534523 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:23.535191 (+ 668us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:23.535537 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:23.536106 (+ 569us) server_negotiation.cc:299] Negotiation successful
0810 11:37:23.536577 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":865,"thread_start_us":501,"threads_started":1}
I20260810 11:37:23.542114 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:37:23.542693 8378 raft_consensus.cc:348] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } }
I20260810 11:37:23.543167 8378 raft_consensus.cc:374] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:23.543282 8378 raft_consensus.cc:729] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 431d20ac026f470e9a0d9d6ebb1842eb, State: Initialized, Role: FOLLOWER
I20260810 11:37:23.543846 8378 consensus_queue.cc:260] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } }
I20260810 11:37:23.544093 8378 raft_consensus.cc:388] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:23.544253 8378 raft_consensus.cc:482] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:23.544435 8378 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:23.546588 8378 raft_consensus.cc:504] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } }
I20260810 11:37:23.547276 8378 leader_election.cc:302] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 431d20ac026f470e9a0d9d6ebb1842eb; no voters:
I20260810 11:37:23.548144 8378 leader_election.cc:288] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:23.548643 8385 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:23.552006 8378 sys_catalog.cc:564] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:23.553025 8385 raft_consensus.cc:686] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [term 1 LEADER]: Becoming Leader. State: Replica: 431d20ac026f470e9a0d9d6ebb1842eb, State: Running, Role: LEADER
I20260810 11:37:23.553695 8385 consensus_queue.cc:237] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } }
I20260810 11:37:23.559893 8387 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 431d20ac026f470e9a0d9d6ebb1842eb. Latest consensus state: current_term: 1 leader_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } } }
I20260810 11:37:23.560374 8387 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:23.560586 8386 sys_catalog.cc:455] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "431d20ac026f470e9a0d9d6ebb1842eb" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41867 } } }
I20260810 11:37:23.560909 8386 sys_catalog.cc:458] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:23.571483 8404 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:23.577792 8404 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:23.590081 8404 catalog_manager.cc:1347] Generated new cluster ID: 2eeaf6d3e3c34412b5a30f06f0966016
I20260810 11:37:23.590248 8404 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:23.612643 8404 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:23.614419 8404 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:23.616006 8383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:23.605675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56050 (local address 127.3.240.254:41867)
0810 11:37:23.605856 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:23.605862 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:23.606481 (+ 619us) server_negotiation.cc:407] Connection header received
0810 11:37:23.606654 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:23.606660 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:23.606751 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:23.606880 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:23.608374 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:23.609795 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:23.611521 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:23.612293 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:23.612437 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:23.614003 (+ 1566us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:23.614010 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:23.614016 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:23.615056 (+ 1040us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:23.615122 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:23.615153 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:23.615280 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:23.615473 (+ 193us) server_negotiation.cc:299] Negotiation successful
0810 11:37:23.615615 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:37:23.625196 8404 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 431d20ac026f470e9a0d9d6ebb1842eb: Generated new TSK 0
I20260810 11:37:23.626608 8404 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:37:23.640021 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8288
I20260810 11:37:23.641304 8370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:23.743599 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8288
2026-08-10T11:37:23Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2335 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:23 dist-test-slave-drl0 krb5kdc[8424](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:23 dist-test-slave-drl0 krb5kdc[8424](info): set up 2 sockets
Aug 10 11:37:23 dist-test-slave-drl0 krb5kdc[8424](info): commencing operation
krb5kdc: starting...
W20260810 11:37:25.109333 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.003s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:25 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:25.379768 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38421
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38421
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:25.761108 8444 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:25.761485 8444 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:25.761656 8444 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:25.769515 8444 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:25.769773 8444 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:25.769901 8444 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:25.770023 8444 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:25.770110 8444 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:25.782454 8444 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38421
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38421
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:25.783981 8444 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:25.785632 8444 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:25.800505 8450 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:25.800551 8451 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:25.800577 8454 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:25.801934 8444 server_base.cc:1034] running on GCE node
I20260810 11:37:25.802799 8444 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:25.804307 8444 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:25.805555 8444 hybrid_clock.cc:648] HybridClock initialized: now 1786361845805509 us; error 37 us; skew 500 ppm
I20260810 11:37:25.805907 8444 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:25 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361845, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:25.813731 8444 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:25.815495 8444 webserver.cc:460] Webserver started at http://127.3.240.254:41681/ using document root <none> and password file <none>
I20260810 11:37:25.816159 8444 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:25.816267 8444 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:25.816521 8444 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:25.819630 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "64b470b8bc1540dbbe0eb2e45289f12f"
format_stamp: "Formatted at 2026-08-10 11:37:25 on dist-test-slave-drl0"
server_key: "b6e525406ee40e3472348ff62ed35110"
server_key_iv: "564552f704b804acbb6c0d17a1f92e66"
server_key_version: "encryptionkey@0"
I20260810 11:37:25.820338 8444 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "64b470b8bc1540dbbe0eb2e45289f12f"
format_stamp: "Formatted at 2026-08-10 11:37:25 on dist-test-slave-drl0"
server_key: "b6e525406ee40e3472348ff62ed35110"
server_key_iv: "564552f704b804acbb6c0d17a1f92e66"
server_key_version: "encryptionkey@0"
I20260810 11:37:25.825620 8444 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 11:37:25.829388 8462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:25.831000 8444 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260810 11:37:25.831218 8444 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "64b470b8bc1540dbbe0eb2e45289f12f"
format_stamp: "Formatted at 2026-08-10 11:37:25 on dist-test-slave-drl0"
server_key: "b6e525406ee40e3472348ff62ed35110"
server_key_iv: "564552f704b804acbb6c0d17a1f92e66"
server_key_version: "encryptionkey@0"
I20260810 11:37:25.831408 8444 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:25.849539 8444 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:25.854111 8444 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:25.854545 8444 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:25.876235 8444 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38421
I20260810 11:37:25.876376 8542 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38421 every 8 connection(s)
I20260810 11:37:25.877723 8444 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:25.881678 8543 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:25.887414 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8444
I20260810 11:37:25.887650 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:25.888348 4035 external_mini_cluster.cc:1428] Setting key 9ccf0f6a44ce241e581ea5dc04f97b3a
I20260810 11:37:25.895469 8543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f: Bootstrap starting.
I20260810 11:37:25.899483 8543 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:25.900722 8543 log.cc:824] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:37:25 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361845, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:25.904649 8543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f: No bootstrap required, opened a new log
I20260810 11:37:25.912704 8543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } }
I20260810 11:37:25.912976 8548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:25.890578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35570 (local address 127.3.240.254:38421)
0810 11:37:25.891884 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:25.891896 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:25.891940 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:37:25.893764 (+ 1824us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:25.893786 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:25.894163 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:25.894479 (+ 316us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:25.896510 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:25.897825 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:25.899758 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:25.900457 (+ 699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:25.900597 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:25.905637 (+ 5040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:25.905645 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:25.905660 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:25.905699 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:25.908920 (+ 3221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:25.909678 (+ 758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:25.909685 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:25.909691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:25.909869 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:25.910273 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:25.910277 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:25.910279 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:25.910827 (+ 548us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:25.911226 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:25.911611 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 11:37:25.912031 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1066,"thread_start_us":602,"threads_started":1}
I20260810 11:37:25.913142 8543 raft_consensus.cc:374] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:25.913348 8543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 64b470b8bc1540dbbe0eb2e45289f12f, State: Initialized, Role: FOLLOWER
I20260810 11:37:25.914062 8543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } }
I20260810 11:37:25.914356 8543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:25.914515 8543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:25.914696 8543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:25.917830 8543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } }
I20260810 11:37:25.918365 8543 leader_election.cc:302] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 64b470b8bc1540dbbe0eb2e45289f12f; no voters:
I20260810 11:37:25.919221 8543 leader_election.cc:288] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:25.919553 8551 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:25.920276 8551 raft_consensus.cc:686] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [term 1 LEADER]: Becoming Leader. State: Replica: 64b470b8bc1540dbbe0eb2e45289f12f, State: Running, Role: LEADER
I20260810 11:37:25.920941 8551 consensus_queue.cc:237] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } }
I20260810 11:37:25.922111 8543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:25.923960 8552 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 64b470b8bc1540dbbe0eb2e45289f12f. Latest consensus state: current_term: 1 leader_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } } }
I20260810 11:37:25.923942 8553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "64b470b8bc1540dbbe0eb2e45289f12f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38421 } } }
I20260810 11:37:25.924351 8552 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:25.924351 8553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:25.925222 8558 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:25.931037 8558 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:25.941895 8558 catalog_manager.cc:1347] Generated new cluster ID: ecb69f2a953b47858eab4492b5ddfa25
I20260810 11:37:25.942009 8558 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:25.972230 8558 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:25.973918 8558 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:25.984369 8558 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 64b470b8bc1540dbbe0eb2e45289f12f: Generated new TSK 0
I20260810 11:37:25.985415 8558 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:26.075085 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:26.452931 8583 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:26.453222 8583 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:26.453379 8583 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:26.461462 8583 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:26.461591 8583 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:26.461712 8583 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:26.471737 8583 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:26.473078 8583 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:26.474424 8583 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:26.489984 8593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:26.489939 8594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:26.490062 8596 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:26.490914 8583 server_base.cc:1034] running on GCE node
I20260810 11:37:26.491727 8583 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:26.492712 8583 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:26.493981 8583 hybrid_clock.cc:648] HybridClock initialized: now 1786361846493927 us; error 44 us; skew 500 ppm
I20260810 11:37:26.494341 8583 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:26 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361846, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:26.497749 8583 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:26.499571 8583 webserver.cc:460] Webserver started at http://127.3.240.193:42259/ using document root <none> and password file <none>
I20260810 11:37:26.500698 8583 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:26.500939 8583 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:26.501385 8583 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:26.505357 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "c3dda6bee71748908c117b5fb90ab142"
format_stamp: "Formatted at 2026-08-10 11:37:26 on dist-test-slave-drl0"
server_key: "921ca1ab419ab7d4e7c4c55130d631e6"
server_key_iv: "ab7523cca24ccef39b2cde352a8afd92"
server_key_version: "encryptionkey@0"
I20260810 11:37:26.506265 8583 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "c3dda6bee71748908c117b5fb90ab142"
format_stamp: "Formatted at 2026-08-10 11:37:26 on dist-test-slave-drl0"
server_key: "921ca1ab419ab7d4e7c4c55130d631e6"
server_key_iv: "ab7523cca24ccef39b2cde352a8afd92"
server_key_version: "encryptionkey@0"
I20260810 11:37:26.511878 8583 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:37:26.515599 8603 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:26.517752 8583 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:26.518098 8583 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "c3dda6bee71748908c117b5fb90ab142"
format_stamp: "Formatted at 2026-08-10 11:37:26 on dist-test-slave-drl0"
server_key: "921ca1ab419ab7d4e7c4c55130d631e6"
server_key_iv: "ab7523cca24ccef39b2cde352a8afd92"
server_key_version: "encryptionkey@0"
I20260810 11:37:26.518373 8583 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:26.544593 8583 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:26.549983 8583 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:26.550349 8583 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:26.551469 8583 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:26.552870 8583 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:26.552982 8583 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:26.553139 8583 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:26.553215 8583 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:26.594527 8583 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38399
I20260810 11:37:26.594616 8761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38399 every 8 connection(s)
I20260810 11:37:26.596176 8583 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:26.603396 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8583
I20260810 11:37:26.603690 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:26.604707 4035 external_mini_cluster.cc:1428] Setting key b8368b816bb09dfecdeeef7b1afc1bcc
Aug 10 11:37:26 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361846, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:26.620817 8765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:26.599127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:32871 (local address 127.3.240.254:38421)
0810 11:37:26.599605 (+ 478us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:26.599610 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:26.601152 (+ 1542us) server_negotiation.cc:407] Connection header received
0810 11:37:26.603824 (+ 2672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:26.603834 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:26.603924 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:26.604032 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:26.606440 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:26.607297 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:26.609765 (+ 2468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:26.610165 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:26.610272 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:26.614748 (+ 4476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:26.614753 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:26.614757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:26.614790 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:26.617457 (+ 2667us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:26.618218 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:26.618224 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:26.618228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:26.618358 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:26.618790 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:26.618796 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:26.618799 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:26.619132 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:26.619288 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:26.620215 (+ 927us) server_negotiation.cc:299] Negotiation successful
0810 11:37:26.620460 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":256,"threads_started":1}
I20260810 11:37:26.622267 8766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:26.599863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38421 (local address 127.3.240.193:32871)
0810 11:37:26.600871 (+ 1008us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:26.600932 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:26.603331 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:26.604419 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:26.604435 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:26.605548 (+ 1113us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:26.606218 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:26.606250 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:26.607491 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:26.607495 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:26.609554 (+ 2059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:26.609574 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:26.610414 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:26.610423 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:26.610745 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:26.611589 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:26.611623 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:26.614518 (+ 2895us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:26.617620 (+ 3102us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:26.617629 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:26.617646 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:26.618092 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:26.618474 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:26.618479 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:26.618483 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:26.618670 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:26.619266 (+ 596us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:26.619274 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:26.619627 (+ 353us) client_negotiation.cc:769] Sending connection context
0810 11:37:26.619920 (+ 293us) client_negotiation.cc:240] Negotiation successful
0810 11:37:26.620574 (+ 654us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":352,"threads_started":1}
I20260810 11:37:26.624686 8762 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38421
I20260810 11:37:26.625059 8762 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:26.625833 8762 heartbeater.cc:511] Master 127.3.240.254:38421 requested a full tablet report, sending...
I20260810 11:37:26.628216 8486 ts_manager.cc:194] Registered new tserver with Master: c3dda6bee71748908c117b5fb90ab142 (127.3.240.193:38399)
I20260810 11:37:26.629676 8486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:32871
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:26.667693 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:27.016422 8769 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:27.016698 8769 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:27.016795 8769 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:27.024547 8769 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:27.024663 8769 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:27.024791 8769 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:27.034789 8769 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:27.036221 8769 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:27.037550 8769 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:27.051162 8779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:27.051370 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:27.051609 8778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:27.051863 8769 server_base.cc:1034] running on GCE node
I20260810 11:37:27.052505 8769 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:27.053524 8769 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:27.054776 8769 hybrid_clock.cc:648] HybridClock initialized: now 1786361847054729 us; error 58 us; skew 500 ppm
I20260810 11:37:27.055158 8769 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:27.058389 8769 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:27.059917 8769 webserver.cc:460] Webserver started at http://127.3.240.194:35103/ using document root <none> and password file <none>
I20260810 11:37:27.060607 8769 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:27.060760 8769 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:27.061089 8769 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:27.064067 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "7f4a0ab849a24da09aa48cbe33abe85d"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "4a248950cb4fd934eb9c8c0772244f18"
server_key_iv: "eee84c2120b9daebbe57927ac4c7c9ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.064853 8769 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "7f4a0ab849a24da09aa48cbe33abe85d"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "4a248950cb4fd934eb9c8c0772244f18"
server_key_iv: "eee84c2120b9daebbe57927ac4c7c9ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.070096 8769 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:27.074290 8794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.075915 8769 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:27.076116 8769 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "7f4a0ab849a24da09aa48cbe33abe85d"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "4a248950cb4fd934eb9c8c0772244f18"
server_key_iv: "eee84c2120b9daebbe57927ac4c7c9ab"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.076334 8769 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:27.096064 8769 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:27.101580 8769 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:27.101924 8769 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:27.103276 8769 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:27.105664 8769 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:27.105791 8769 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.105957 8769 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:27.106041 8769 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.147451 8769 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:44327
I20260810 11:37:27.147538 8960 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:44327 every 8 connection(s)
I20260810 11:37:27.148754 8769 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:27.154429 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8769
I20260810 11:37:27.154639 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:27.155710 4035 external_mini_cluster.cc:1428] Setting key 600ea37ae165f31ec1b6a62d580e6532
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:27.170894 8963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.151721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:36195 (local address 127.3.240.254:38421)
0810 11:37:27.152433 (+ 712us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.152441 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.153397 (+ 956us) server_negotiation.cc:407] Connection header received
0810 11:37:27.155540 (+ 2143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.155546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.155633 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.155766 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.157662 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.158394 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.160022 (+ 1628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.160475 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.160612 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.164930 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.164936 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.164939 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.164970 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.167819 (+ 2849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.168556 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.168563 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.168572 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.168693 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.169153 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.169158 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.169161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.169419 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.169586 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.170264 (+ 678us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.170483 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":544,"thread_start_us":397,"threads_started":1}
I20260810 11:37:27.171869 8964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.152363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38421 (local address 127.3.240.194:36195)
0810 11:37:27.153236 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:27.153282 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:27.155233 (+ 1951us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:27.156018 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:27.156025 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:27.156680 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:27.157475 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:27.157503 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.158571 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.158575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:27.159869 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:27.159883 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.160714 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.160722 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:27.160962 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.162418 (+ 1456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:27.162453 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:27.164690 (+ 2237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:27.167968 (+ 3278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:27.167977 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:27.167993 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:27.168418 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:27.168810 (+ 392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:27.168818 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:27.168822 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:27.169018 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:27.169552 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:27.169563 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:27.169897 (+ 334us) client_negotiation.cc:769] Sending connection context
0810 11:37:27.170174 (+ 277us) client_negotiation.cc:240] Negotiation successful
0810 11:37:27.170701 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":393,"threads_started":1}
I20260810 11:37:27.173455 8961 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38421
I20260810 11:37:27.173775 8961 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:27.174525 8961 heartbeater.cc:511] Master 127.3.240.254:38421 requested a full tablet report, sending...
I20260810 11:37:27.176319 8486 ts_manager.cc:194] Registered new tserver with Master: 7f4a0ab849a24da09aa48cbe33abe85d (127.3.240.194:44327)
I20260810 11:37:27.177659 8486 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:36195
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:27.211611 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39837
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:27.564394 8967 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:27.564656 8967 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:27.564754 8967 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:27.572183 8967 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:27.572286 8967 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:27.572398 8967 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:27.582540 8967 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39837
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38421
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:27.584373 8967 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:27.585685 8967 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:27.600780 8973 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:27.600800 8975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:27.601186 8977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:27.602022 8967 server_base.cc:1034] running on GCE node
I20260810 11:37:27.602630 8967 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:27.603713 8967 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:27.604964 8967 hybrid_clock.cc:648] HybridClock initialized: now 1786361847604905 us; error 45 us; skew 500 ppm
I20260810 11:37:27.605325 8967 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:27.608839 8967 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:27.610496 8967 webserver.cc:460] Webserver started at http://127.3.240.195:37407/ using document root <none> and password file <none>
I20260810 11:37:27.611335 8967 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:27.611496 8967 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:27.611810 8967 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:27.614714 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "57785c59a47e41a8b220e22523bf27aa"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "8096a0421133444c73b213b06f08bc51"
server_key_iv: "0ac93cd2e1dfa8574cfba7690ad1d14a"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.615584 8967 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "57785c59a47e41a8b220e22523bf27aa"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "8096a0421133444c73b213b06f08bc51"
server_key_iv: "0ac93cd2e1dfa8574cfba7690ad1d14a"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.621140 8967 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 11:37:27.625356 8988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.626967 8967 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:37:27.627159 8967 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "57785c59a47e41a8b220e22523bf27aa"
format_stamp: "Formatted at 2026-08-10 11:37:27 on dist-test-slave-drl0"
server_key: "8096a0421133444c73b213b06f08bc51"
server_key_iv: "0ac93cd2e1dfa8574cfba7690ad1d14a"
server_key_version: "encryptionkey@0"
I20260810 11:37:27.627378 8967 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:27.633391 8762 heartbeater.cc:503] Master 127.3.240.254:38421 was elected leader, sending a full tablet report...
I20260810 11:37:27.647409 8967 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:27.652109 8967 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:27.652444 8967 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:27.653443 8967 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:27.654819 8967 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:27.654896 8967 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.655079 8967 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:27.655171 8967 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:27.695341 8967 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41773
I20260810 11:37:27.695405 9141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41773 every 8 connection(s)
I20260810 11:37:27.696641 8967 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:27.706686 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 8967
I20260810 11:37:27.707007 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:27.707512 4035 external_mini_cluster.cc:1428] Setting key aabc8a683b196e665998399a4522967b
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:27.720680 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.700185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:58135 (local address 127.3.240.254:38421)
0810 11:37:27.700799 (+ 614us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.700806 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.702222 (+ 1416us) server_negotiation.cc:407] Connection header received
0810 11:37:27.704103 (+ 1881us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.704108 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.704198 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.704304 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.706094 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.707019 (+ 925us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.708750 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.709385 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.709540 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.713448 (+ 3908us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.713453 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.713457 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.713489 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.716351 (+ 2862us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.717329 (+ 978us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.717333 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.717335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.717476 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.718499 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.718505 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.718508 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.718920 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.719267 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.720062 (+ 795us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.720329 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":368,"threads_started":1}
I20260810 11:37:27.721382 9148 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.700887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38421 (local address 127.3.240.195:58135)
0810 11:37:27.702004 (+ 1117us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:27.702049 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:27.703810 (+ 1761us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:27.704607 (+ 797us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:27.704615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:27.705216 (+ 601us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:27.705902 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:27.705924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.707205 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.707210 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:27.708571 (+ 1361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:27.708588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.709674 (+ 1086us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.709682 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:27.710107 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.711032 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:27.711072 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:27.713225 (+ 2153us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:27.716502 (+ 3277us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:27.716508 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:27.716520 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:27.716946 (+ 426us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:27.717728 (+ 782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:27.717734 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:27.717738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:27.718218 (+ 480us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:27.719263 (+ 1045us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:27.719270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:27.719603 (+ 333us) client_negotiation.cc:769] Sending connection context
0810 11:37:27.719859 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 11:37:27.720243 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"thread_start_us":503,"threads_started":1}
I20260810 11:37:27.722968 9142 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38421
I20260810 11:37:27.723344 9142 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:27.724365 9142 heartbeater.cc:511] Master 127.3.240.254:38421 requested a full tablet report, sending...
I20260810 11:37:27.726027 8483 ts_manager.cc:194] Registered new tserver with Master: 57785c59a47e41a8b220e22523bf27aa (127.3.240.195:41773)
I20260810 11:37:27.727176 8483 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:58135
I20260810 11:37:27.737217 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:27.798167 9146 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.781959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35586 (local address 127.3.240.254:38421)
0810 11:37:27.782232 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.782238 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.782535 (+ 297us) server_negotiation.cc:407] Connection header received
0810 11:37:27.782653 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.782657 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.782739 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.782826 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.784331 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.785415 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.786816 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.787365 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.787484 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.791558 (+ 4074us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.791563 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.791567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.791602 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.794723 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.795836 (+ 1113us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.795846 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.795849 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.795986 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.796481 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.796485 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.796487 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.796905 (+ 418us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.797131 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.797666 (+ 535us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.797870 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
W20260810 11:37:27.798853 8483 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:35586
I20260810 11:37:27.801468 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8583
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:27.819556 9158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.802617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39372 (local address 127.3.240.193:38399)
0810 11:37:27.803673 (+ 1056us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.803681 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.803709 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:37:27.803781 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.803786 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.803999 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.804132 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.805421 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.806608 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.808078 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.808712 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.808828 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.812930 (+ 4102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.812942 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.812957 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.812989 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.816427 (+ 3438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.817137 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.817145 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.817148 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.817294 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.817793 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.817799 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.817802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.818336 (+ 534us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.818599 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.819019 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.819237 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":788,"thread_start_us":568,"threads_started":1}
W20260810 11:37:27.821245 8756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:39372
W20260810 11:37:27.822768 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8583: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:27.823122 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8583
I20260810 11:37:27.847679 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8769
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:27.866274 9160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.848702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49988 (local address 127.3.240.194:44327)
0810 11:37:27.849227 (+ 525us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.849246 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.849272 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:37:27.849333 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.849340 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.849555 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.849923 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.851168 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.852232 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.853763 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.854346 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.854457 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.859368 (+ 4911us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.859392 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.859416 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.859472 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.863152 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.863886 (+ 734us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.863895 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.863898 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.864041 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.864467 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.864472 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.864477 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.864837 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.865195 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.865611 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.865894 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":288,"threads_started":1}
W20260810 11:37:27.867631 8955 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49988
W20260810 11:37:27.868705 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8769: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:27.868835 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8769
I20260810 11:37:27.897302 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8967
Aug 10 11:37:27 dist-test-slave-drl0 krb5kdc[8424](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361847, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:27.918108 9161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:27.899470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40928 (local address 127.3.240.195:41773)
0810 11:37:27.900285 (+ 815us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:27.900293 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:27.900322 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:37:27.900387 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:27.900393 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:27.900843 (+ 450us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:27.901069 (+ 226us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:27.902423 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.903930 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.905600 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:27.906253 (+ 653us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:27.906414 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:27.911148 (+ 4734us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:27.911161 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:27.911177 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:27.911222 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:27.914131 (+ 2909us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.914868 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.914875 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.914878 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.915102 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:27.915859 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:27.915866 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:27.915868 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:27.916426 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:27.916753 (+ 327us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:27.917180 (+ 427us) server_negotiation.cc:299] Negotiation successful
0810 11:37:27.917532 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":655,"thread_start_us":492,"threads_started":1}
W20260810 11:37:27.919602 9134 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:40928
W20260810 11:37:27.920711 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8967: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:37:27.920871 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8967
I20260810 11:37:27.950032 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 8444
I20260810 11:37:27.951529 8537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:28.061146 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 8444
2026-08-10T11:37:28Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4322 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:28 dist-test-slave-drl0 krb5kdc[9166](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:28 dist-test-slave-drl0 krb5kdc[9166](info): set up 2 sockets
Aug 10 11:37:28 dist-test-slave-drl0 krb5kdc[9166](info): commencing operation
krb5kdc: starting...
W20260810 11:37:29.408607 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.294s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:29 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:29.668556 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42229
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42229
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:30.027493 9186 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:30.028079 9186 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:30.028265 9186 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:30.036562 9186 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:30.036705 9186 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:30.036829 9186 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:30.036935 9186 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:30.037021 9186 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:30.046952 9186 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42229
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:30.048271 9186 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:30.049571 9186 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:30.062356 9191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:30.062248 9195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:30.062316 9193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:30.063488 9186 server_base.cc:1034] running on GCE node
I20260810 11:37:30.064287 9186 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:30.065980 9186 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:30.067364 9186 hybrid_clock.cc:648] HybridClock initialized: now 1786361850067279 us; error 108 us; skew 500 ppm
I20260810 11:37:30.067817 9186 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:30 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:30.071110 9186 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:30.072718 9186 webserver.cc:460] Webserver started at http://127.3.240.254:41941/ using document root <none> and password file <none>
I20260810 11:37:30.073423 9186 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:30.073665 9186 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:30.074043 9186 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:30.077208 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "2effc8dffb6e4a31bb9635439c2ff2d2"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "45407e24480450f3ac5a872b87145211"
server_key_iv: "a830fe81cb7399c5f5813d474975f3e2"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.077998 9186 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2effc8dffb6e4a31bb9635439c2ff2d2"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "45407e24480450f3ac5a872b87145211"
server_key_iv: "a830fe81cb7399c5f5813d474975f3e2"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.083674 9186 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:37:30.088260 9203 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:30.089936 9186 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 11:37:30.090162 9186 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2effc8dffb6e4a31bb9635439c2ff2d2"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "45407e24480450f3ac5a872b87145211"
server_key_iv: "a830fe81cb7399c5f5813d474975f3e2"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.090371 9186 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:30.108830 9186 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:30.113505 9186 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:30.113924 9186 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:30.136268 9186 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42229
I20260810 11:37:30.136293 9272 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42229 every 8 connection(s)
I20260810 11:37:30.137705 9186 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:30.142627 9273 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:30.147528 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 9186
I20260810 11:37:30.147934 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:30.148813 4035 external_mini_cluster.cc:1428] Setting key 6f6a540e622e7ad98670ad01ad3e783b
I20260810 11:37:30.156520 9273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2: Bootstrap starting.
I20260810 11:37:30.160064 9273 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:30.161249 9273 log.cc:824] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:37:30 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361849, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:30.165016 9273 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2: No bootstrap required, opened a new log
I20260810 11:37:30.177116 9277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:30.151114 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44624 (local address 127.3.240.254:42229)
0810 11:37:30.152035 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:30.152047 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:30.152089 (+ 42us) server_negotiation.cc:407] Connection header received
0810 11:37:30.154693 (+ 2604us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:30.154717 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:30.155171 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:30.155576 (+ 405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:30.157129 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.158602 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.159924 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.160674 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.160884 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:30.166434 (+ 5550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:30.166443 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:30.166457 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:30.166492 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:30.170229 (+ 3737us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:30.172587 (+ 2358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:30.172598 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:30.172609 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:30.172935 (+ 326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:30.173622 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:30.173627 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:30.173630 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:30.174416 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:30.174754 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:30.175224 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 11:37:30.175679 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":614,"thread_start_us":401,"threads_started":1}
I20260810 11:37:30.178427 9273 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } }
I20260810 11:37:30.178906 9273 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:30.179065 9273 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2effc8dffb6e4a31bb9635439c2ff2d2, State: Initialized, Role: FOLLOWER
I20260810 11:37:30.179687 9273 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } }
I20260810 11:37:30.180086 9273 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:30.180271 9273 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:30.180497 9273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:30.183071 9273 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } }
I20260810 11:37:30.183676 9273 leader_election.cc:302] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2effc8dffb6e4a31bb9635439c2ff2d2; no voters:
I20260810 11:37:30.184319 9273 leader_election.cc:288] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:30.184626 9279 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:30.185673 9279 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [term 1 LEADER]: Becoming Leader. State: Replica: 2effc8dffb6e4a31bb9635439c2ff2d2, State: Running, Role: LEADER
I20260810 11:37:30.186240 9279 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } }
I20260810 11:37:30.186690 9273 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:30.189393 9280 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } } }
I20260810 11:37:30.189772 9280 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:30.189488 9281 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2effc8dffb6e4a31bb9635439c2ff2d2. Latest consensus state: current_term: 1 leader_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2effc8dffb6e4a31bb9635439c2ff2d2" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42229 } } }
I20260810 11:37:30.189920 9281 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:30.192946 9287 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:30.197880 9287 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:30.207898 9287 catalog_manager.cc:1347] Generated new cluster ID: 8b17fd86cf4f4f18a14f3b4694f4b1ec
I20260810 11:37:30.208024 9287 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:30.216996 9287 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:30.219271 9287 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:30.234725 9287 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2effc8dffb6e4a31bb9635439c2ff2d2: Generated new TSK 0
I20260810 11:37:30.236136 9287 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:30.307087 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:30.665077 9307 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:30.665360 9307 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:30.665467 9307 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:30.672747 9307 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:30.672861 9307 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:30.672976 9307 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:30.687357 9307 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:30.688845 9307 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:30.690127 9307 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:30.705689 9317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:30.705756 9319 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:30.705788 9315 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:30.708567 9307 server_base.cc:1034] running on GCE node
I20260810 11:37:30.709533 9307 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:30.710662 9307 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:30.711946 9307 hybrid_clock.cc:648] HybridClock initialized: now 1786361850711880 us; error 50 us; skew 500 ppm
I20260810 11:37:30.712379 9307 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:30 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:30.716056 9307 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:30.717734 9307 webserver.cc:460] Webserver started at http://127.3.240.193:42301/ using document root <none> and password file <none>
I20260810 11:37:30.718489 9307 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:30.718649 9307 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:30.719086 9307 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:30.722842 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "d924f99613db46998f954d2b39ca961f"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "8712eb8b1292d7275b68264bc41c4207"
server_key_iv: "99fee54338eb18d98c2300e7d90e4255"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.724201 9307 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "d924f99613db46998f954d2b39ca961f"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "8712eb8b1292d7275b68264bc41c4207"
server_key_iv: "99fee54338eb18d98c2300e7d90e4255"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.729655 9307 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 11:37:30.733683 9329 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:30.735255 9307 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:30.735467 9307 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "d924f99613db46998f954d2b39ca961f"
format_stamp: "Formatted at 2026-08-10 11:37:30 on dist-test-slave-drl0"
server_key: "8712eb8b1292d7275b68264bc41c4207"
server_key_iv: "99fee54338eb18d98c2300e7d90e4255"
server_key_version: "encryptionkey@0"
I20260810 11:37:30.735693 9307 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:30.764616 9307 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:30.769260 9307 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:30.769578 9307 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:30.770586 9307 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:30.772110 9307 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:30.772218 9307 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:30.772380 9307 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:30.772455 9307 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:30.814031 9307 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44869
I20260810 11:37:30.814072 9492 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44869 every 8 connection(s)
I20260810 11:37:30.815361 9307 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:30.822611 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 9307
I20260810 11:37:30.822865 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:30.823755 4035 external_mini_cluster.cc:1428] Setting key ad38c1a138b8fd0d71420c61ee36682d
Aug 10 11:37:30 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361850, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:30.845605 9498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:30.818394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56171 (local address 127.3.240.254:42229)
0810 11:37:30.819086 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:30.819092 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:30.820532 (+ 1440us) server_negotiation.cc:407] Connection header received
0810 11:37:30.822944 (+ 2412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:30.822952 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:30.823078 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:30.823182 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:30.825373 (+ 2191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.826407 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.829165 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.829575 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.829682 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:30.837618 (+ 7936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:30.837627 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:30.837636 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:30.837684 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:30.841778 (+ 4094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:30.842621 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:30.842630 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:30.842637 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:30.842825 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:30.843389 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:30.843397 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:30.843403 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:30.843849 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:30.844021 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:30.844834 (+ 813us) server_negotiation.cc:299] Negotiation successful
0810 11:37:30.845103 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":358,"threads_started":1}
I20260810 11:37:30.846269 9499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:30.819386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42229 (local address 127.3.240.193:56171)
0810 11:37:30.820326 (+ 940us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:30.820370 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:30.822536 (+ 2166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:30.823500 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:30.823513 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:30.824274 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:30.825180 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:30.825201 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.826621 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.826627 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:30.828941 (+ 2314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:30.828972 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:30.829838 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:30.829848 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:30.830192 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:30.834512 (+ 4320us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:30.834548 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:30.837340 (+ 2792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:30.841963 (+ 4623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:30.841971 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:30.841993 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:30.842451 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:30.843034 (+ 583us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:30.843037 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:30.843040 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:30.843246 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:30.844018 (+ 772us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:30.844024 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:30.844386 (+ 362us) client_negotiation.cc:769] Sending connection context
0810 11:37:30.844633 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 11:37:30.845054 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":666,"thread_start_us":401,"threads_started":1}
I20260810 11:37:30.848188 9494 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42229
I20260810 11:37:30.848536 9494 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:30.849292 9494 heartbeater.cc:511] Master 127.3.240.254:42229 requested a full tablet report, sending...
I20260810 11:37:30.851536 9223 ts_manager.cc:194] Registered new tserver with Master: d924f99613db46998f954d2b39ca961f (127.3.240.193:44869)
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:37:30.853114 9223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:56171
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:30.888374 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:31.256603 9502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:31.257048 9502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:31.257234 9502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:31.264986 9502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:31.265139 9502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:31.265318 9502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:31.274998 9502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:31.276305 9502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:31.277611 9502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:31.290768 9513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:31.290791 9509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:31.290915 9511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:31.292346 9502 server_base.cc:1034] running on GCE node
I20260810 11:37:31.292975 9502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:31.293926 9502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:31.295189 9502 hybrid_clock.cc:648] HybridClock initialized: now 1786361851295143 us; error 43 us; skew 500 ppm
I20260810 11:37:31.295579 9502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:31 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:31.299494 9502 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:31.301079 9502 webserver.cc:460] Webserver started at http://127.3.240.194:40625/ using document root <none> and password file <none>
I20260810 11:37:31.301786 9502 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:31.301949 9502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:31.302271 9502 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:31.305778 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "6d364fbae5214cd5a143bfc827a13f43"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "991fae31319690cb910afe05160bc5cb"
server_key_iv: "c1fe6072f1399400274d3b63da2feaab"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.306598 9502 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "6d364fbae5214cd5a143bfc827a13f43"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "991fae31319690cb910afe05160bc5cb"
server_key_iv: "c1fe6072f1399400274d3b63da2feaab"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.312264 9502 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.000s
I20260810 11:37:31.316310 9520 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.318451 9502 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 11:37:31.318764 9502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "6d364fbae5214cd5a143bfc827a13f43"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "991fae31319690cb910afe05160bc5cb"
server_key_iv: "c1fe6072f1399400274d3b63da2feaab"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.319145 9502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:31.337292 9502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:31.342443 9502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:31.343019 9502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:31.344165 9502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:31.345551 9502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:31.345659 9502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.345818 9502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:31.345896 9502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.384963 9502 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39815
I20260810 11:37:31.385056 9687 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39815 every 8 connection(s)
I20260810 11:37:31.386265 9502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:31.396566 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 9502
I20260810 11:37:31.396770 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:31.397231 4035 external_mini_cluster.cc:1428] Setting key b335841b1bbcbae1bb20d42f3c21efe1
Aug 10 11:37:31 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:31.410213 9690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:31.389080 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34995 (local address 127.3.240.254:42229)
0810 11:37:31.389595 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:31.389600 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:31.390372 (+ 772us) server_negotiation.cc:407] Connection header received
0810 11:37:31.392285 (+ 1913us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:31.392292 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:31.392397 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:31.392558 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:31.394152 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.395044 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.396893 (+ 1849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.397387 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.397507 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:31.401527 (+ 4020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:31.401537 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:31.401545 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:31.401592 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:31.405774 (+ 4182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:31.406524 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:31.406532 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:31.406537 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:31.406803 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:31.407334 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:31.407342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:31.407348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:31.408053 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:31.408432 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:31.409250 (+ 818us) server_negotiation.cc:299] Negotiation successful
0810 11:37:31.409705 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":284,"threads_started":1}
I20260810 11:37:31.410543 9691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:31.389450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42229 (local address 127.3.240.194:34995)
0810 11:37:31.390196 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:31.390237 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:31.391995 (+ 1758us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:31.392790 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:31.392798 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:31.393312 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:31.393988 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:31.394007 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.395209 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.395213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:31.396674 (+ 1461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:31.396690 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.397628 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.397636 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:31.397915 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:31.398760 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:31.398788 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:31.401286 (+ 2498us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:31.405939 (+ 4653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:31.405946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:31.405959 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:31.406398 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:31.407009 (+ 611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:31.407013 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:31.407015 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:31.407205 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:31.408387 (+ 1182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:31.408393 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:31.408714 (+ 321us) client_negotiation.cc:769] Sending connection context
0810 11:37:31.408961 (+ 247us) client_negotiation.cc:240] Negotiation successful
0810 11:37:31.409375 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":497,"thread_start_us":290,"threads_started":1}
I20260810 11:37:31.412225 9688 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42229
I20260810 11:37:31.412773 9688 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:31.414036 9688 heartbeater.cc:511] Master 127.3.240.254:42229 requested a full tablet report, sending...
I20260810 11:37:31.416194 9223 ts_manager.cc:194] Registered new tserver with Master: 6d364fbae5214cd5a143bfc827a13f43 (127.3.240.194:39815)
I20260810 11:37:31.417132 9223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:34995
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:31.462141 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:31.817375 9695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:31.817688 9695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:31.817870 9695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:31.826100 9695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:31.826264 9695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:31.826450 9695 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:31.836414 9695 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42229
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:31.837906 9695 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:31.839363 9695 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:31.853248 9706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:31.853482 9704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:31.854240 9703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:31.854735 9695 server_base.cc:1034] running on GCE node
I20260810 11:37:31.855443 9695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:31.856521 9695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:31.856864 9494 heartbeater.cc:503] Master 127.3.240.254:42229 was elected leader, sending a full tablet report...
I20260810 11:37:31.857949 9695 hybrid_clock.cc:648] HybridClock initialized: now 1786361851857878 us; error 45 us; skew 500 ppm
I20260810 11:37:31.858346 9695 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:31 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:31.862128 9695 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:31.863746 9695 webserver.cc:460] Webserver started at http://127.3.240.195:39015/ using document root <none> and password file <none>
I20260810 11:37:31.864421 9695 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:31.864560 9695 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:31.864903 9695 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:31.868172 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "1cb230752eb04073bd5b3d369f8126c4"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "95edd85f40dfa0f337969bbc6568d825"
server_key_iv: "a18dc41717f6912b64766aca73431565"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.868960 9695 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "1cb230752eb04073bd5b3d369f8126c4"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "95edd85f40dfa0f337969bbc6568d825"
server_key_iv: "a18dc41717f6912b64766aca73431565"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.874312 9695 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260810 11:37:31.878404 9716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.879921 9695 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:31.880122 9695 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "1cb230752eb04073bd5b3d369f8126c4"
format_stamp: "Formatted at 2026-08-10 11:37:31 on dist-test-slave-drl0"
server_key: "95edd85f40dfa0f337969bbc6568d825"
server_key_iv: "a18dc41717f6912b64766aca73431565"
server_key_version: "encryptionkey@0"
I20260810 11:37:31.880347 9695 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:31.899664 9695 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:31.905400 9695 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:31.906266 9695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:31.907440 9695 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:31.908981 9695 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:31.909106 9695 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.909269 9695 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:31.909353 9695 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:31.962951 9695 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40713
I20260810 11:37:31.962988 9875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40713 every 8 connection(s)
I20260810 11:37:31.964860 9695 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:31.969758 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 9695
I20260810 11:37:31.969985 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:31.970613 4035 external_mini_cluster.cc:1428] Setting key bfc7f2756af58ad91dbcb1964f42f20f
Aug 10 11:37:31 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361851, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:31.991137 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:31.969473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35483 (local address 127.3.240.254:42229)
0810 11:37:31.970078 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:31.970084 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:31.971640 (+ 1556us) server_negotiation.cc:407] Connection header received
0810 11:37:31.974219 (+ 2579us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:31.974223 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:31.974363 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:31.974476 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:31.976981 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.977783 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.979593 (+ 1810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.979973 (+ 380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.980084 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:31.984011 (+ 3927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:31.984017 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:31.984022 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:31.984064 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:31.987385 (+ 3321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:31.988249 (+ 864us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:31.988253 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:31.988255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:31.988379 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:31.988974 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:31.988982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:31.988985 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:31.989415 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:31.989624 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:31.990401 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 11:37:31.990646 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":348,"threads_started":1}
I20260810 11:37:31.992252 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:31.970252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42229 (local address 127.3.240.195:35483)
0810 11:37:31.971468 (+ 1216us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:31.971523 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:31.973919 (+ 2396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:31.975292 (+ 1373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:31.975310 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:31.975946 (+ 636us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:31.976715 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:31.976751 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.977995 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.978000 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:31.979431 (+ 1431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:31.979447 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:31.980289 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:31.980296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:31.980507 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:31.981253 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:31.981279 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:31.983772 (+ 2493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:31.987545 (+ 3773us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:31.987552 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:31.987564 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:31.988102 (+ 538us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:31.988504 (+ 402us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:31.988507 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:31.988510 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:31.988699 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:31.989606 (+ 907us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:31.989612 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:31.989938 (+ 326us) client_negotiation.cc:769] Sending connection context
0810 11:37:31.990204 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 11:37:31.990585 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":929,"thread_start_us":651,"threads_started":1}
I20260810 11:37:31.994455 9876 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42229
I20260810 11:37:31.994808 9876 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:31.995849 9876 heartbeater.cc:511] Master 127.3.240.254:42229 requested a full tablet report, sending...
I20260810 11:37:31.997890 9223 ts_manager.cc:194] Registered new tserver with Master: 1cb230752eb04073bd5b3d369f8126c4 (127.3.240.195:40713)
I20260810 11:37:31.998790 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:37:31.999019 9223 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:35483
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9166](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:32.058693 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:32.041526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39262 (local address 127.3.240.193:44869)
0810 11:37:32.042245 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:32.042252 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:32.042278 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:37:32.042345 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:32.042352 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:32.042574 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:32.042747 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:32.044215 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.045331 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.046526 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.047160 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.047299 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:32.051956 (+ 4657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:32.051967 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:32.051981 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:32.052022 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:32.055083 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.056074 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.056081 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.056083 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.056223 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.056677 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.056681 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.056686 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.057044 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:32.057277 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:32.057695 (+ 418us) server_negotiation.cc:299] Negotiation successful
0810 11:37:32.058049 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":338,"threads_started":1}
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:32.079564 9879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:32.064351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44626 (local address 127.3.240.254:42229)
0810 11:37:32.064647 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:32.064652 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:32.064919 (+ 267us) server_negotiation.cc:407] Connection header received
0810 11:37:32.065144 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:32.065149 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:32.065228 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:32.065359 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:32.066710 (+ 1351us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.067510 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.068935 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.069351 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.069572 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:32.073891 (+ 4319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:32.073896 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:32.073899 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:32.073931 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:32.076736 (+ 2805us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.077383 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.077388 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.077390 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.077515 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.077985 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.077989 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.077991 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.078310 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:32.078537 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:32.078907 (+ 370us) server_negotiation.cc:299] Negotiation successful
0810 11:37:32.079241 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
W20260810 11:37:32.080168 9223 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44626
I20260810 11:37:32.081488 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 9307
I20260810 11:37:32.097450 9889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:32.082715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39278 (local address 127.3.240.193:44869)
0810 11:37:32.083008 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:32.083015 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:32.083208 (+ 193us) server_negotiation.cc:407] Connection header received
0810 11:37:32.083417 (+ 209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:32.083422 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:32.083524 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:32.083656 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:32.085113 (+ 1457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.085855 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.087221 (+ 1366us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.087643 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.087757 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:32.090539 (+ 2782us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:32.090544 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:32.090548 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:32.090580 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:32.093663 (+ 3083us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.094904 (+ 1241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.094908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.094910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.095126 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.095502 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.095505 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.095507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.095944 (+ 437us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:32.096176 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:32.096686 (+ 510us) server_negotiation.cc:299] Negotiation successful
0810 11:37:32.097194 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260810 11:37:32.098138 9487 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:32.187744 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 9307
I20260810 11:37:32.211956 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 9502
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:32.231889 9896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:32.213130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46836 (local address 127.3.240.194:39815)
0810 11:37:32.213888 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:32.213903 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:32.213928 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:37:32.213991 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:32.213997 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:32.214855 (+ 858us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:32.215232 (+ 377us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:32.216726 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.217802 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.219326 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.220037 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.220165 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:32.225842 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:32.225853 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:32.225867 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:32.225911 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:32.228839 (+ 2928us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.229457 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.229464 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.229466 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.229580 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.229940 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.229943 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.229945 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.230271 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:32.230500 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:32.231126 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 11:37:32.231468 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":471,"threads_started":1}
I20260810 11:37:32.233318 9681 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:32.321441 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 9502
I20260810 11:37:32.347652 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 9695
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9166](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361852, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:32.367184 9898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:32.349022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37726 (local address 127.3.240.195:40713)
0810 11:37:32.349908 (+ 886us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:32.349915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:32.349950 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:37:32.350023 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:32.350031 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:32.350265 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:32.350574 (+ 309us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:32.352075 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.353170 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.354522 (+ 1352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:32.355299 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:32.355462 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:32.361139 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:32.361152 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:32.361166 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:32.361199 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:32.364134 (+ 2935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.364974 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.364982 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.364984 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.365112 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:32.365509 (+ 397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:32.365514 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:32.365519 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:32.365883 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:32.366205 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:32.366514 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 11:37:32.366791 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":412,"threads_started":1}
I20260810 11:37:32.368515 9871 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:32.457166 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 9695
I20260810 11:37:32.479202 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 9186
I20260810 11:37:32.480461 9268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:32.592628 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 9186
2026-08-10T11:37:32Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4531 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9905](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9905](info): set up 2 sockets
Aug 10 11:37:32 dist-test-slave-drl0 krb5kdc[9905](info): commencing operation
krb5kdc: starting...
W20260810 11:37:33.949756 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:34 dist-test-slave-drl0 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:34.203110 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39731
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:34.549456 9922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:34.549747 9922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:34.549886 9922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:34.557446 9922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:34.557561 9922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:34.557613 9922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:34.557756 9922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:34.557825 9922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:34.567936 9922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:34.569263 9922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:34.570583 9922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:34.585021 9930 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:34.585084 9929 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:34.585191 9933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:34.585907 9922 server_base.cc:1034] running on GCE node
I20260810 11:37:34.586694 9922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:34.588100 9922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:34.589359 9922 hybrid_clock.cc:648] HybridClock initialized: now 1786361854589295 us; error 64 us; skew 500 ppm
I20260810 11:37:34.589764 9922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:34 dist-test-slave-drl0 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:34.593142 9922 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:34.594511 9922 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:34.594650 9922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:34.595240 9922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:34.598544 9922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "8ddea9e9505c49b68cb5426fe0e8050c"
format_stamp: "Formatted at 2026-08-10 11:37:34 on dist-test-slave-drl0"
server_key: "8130d43929a7da98fc5130a28cc128cc"
server_key_iv: "bc0280de8c03b908bf328552eaec812f"
server_key_version: "encryptionkey@0"
I20260810 11:37:34.599701 9922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "8ddea9e9505c49b68cb5426fe0e8050c"
format_stamp: "Formatted at 2026-08-10 11:37:34 on dist-test-slave-drl0"
server_key: "8130d43929a7da98fc5130a28cc128cc"
server_key_iv: "bc0280de8c03b908bf328552eaec812f"
server_key_version: "encryptionkey@0"
I20260810 11:37:34.606295 9922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260810 11:37:34.610023 9943 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:34.611590 9922 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260810 11:37:34.611799 9922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "8ddea9e9505c49b68cb5426fe0e8050c"
format_stamp: "Formatted at 2026-08-10 11:37:34 on dist-test-slave-drl0"
server_key: "8130d43929a7da98fc5130a28cc128cc"
server_key_iv: "bc0280de8c03b908bf328552eaec812f"
server_key_version: "encryptionkey@0"
I20260810 11:37:34.612047 9922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:34.649835 9922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:34.655150 9922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:34.655442 9922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:34.676823 9922 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39731
I20260810 11:37:34.676841 10016 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39731 every 8 connection(s)
I20260810 11:37:34.677832 9922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:34.681937 10017 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:34.687757 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 9922
I20260810 11:37:34.688002 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:34.688651 4035 external_mini_cluster.cc:1428] Setting key ab1afe13038df0b2d67b1a88a6eb02e6
I20260810 11:37:34.694506 10017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: Bootstrap starting.
I20260810 11:37:34.698218 10017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:34.699987 10017 log.cc:824] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:34.703867 10017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: No bootstrap required, opened a new log
Aug 10 11:37:34 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:34.717052 10017 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } }
I20260810 11:37:34.717494 10021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:34.690772 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60976 (local address 127.3.240.254:39731)
0810 11:37:34.691703 (+ 931us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:34.691726 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:34.691785 (+ 59us) server_negotiation.cc:407] Connection header received
0810 11:37:34.694173 (+ 2388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:34.694199 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:34.694689 (+ 490us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:34.695137 (+ 448us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:34.696542 (+ 1405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:34.698528 (+ 1986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:34.700450 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:34.701606 (+ 1156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:34.701871 (+ 265us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:34.706265 (+ 4394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:34.706281 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:34.706302 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:34.706359 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:34.711538 (+ 5179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:34.712458 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:34.712471 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:34.712487 (+ 16us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:34.712858 (+ 371us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:34.713345 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:34.713351 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:34.713354 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:34.714190 (+ 836us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:34.714539 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:34.715119 (+ 580us) server_negotiation.cc:299] Negotiation successful
0810 11:37:34.716083 (+ 964us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":667,"thread_start_us":415,"threads_started":1}
I20260810 11:37:34.717509 10017 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:34.717684 10017 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8ddea9e9505c49b68cb5426fe0e8050c, State: Initialized, Role: FOLLOWER
I20260810 11:37:34.718315 10017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } }
I20260810 11:37:34.718569 10017 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:34.718721 10017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:34.718922 10017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:34.722052 10017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } }
I20260810 11:37:34.722548 10017 leader_election.cc:302] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8ddea9e9505c49b68cb5426fe0e8050c; no voters:
I20260810 11:37:34.723214 10017 leader_election.cc:288] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:34.723383 10023 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:34.724112 10023 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [term 1 LEADER]: Becoming Leader. State: Replica: 8ddea9e9505c49b68cb5426fe0e8050c, State: Running, Role: LEADER
I20260810 11:37:34.724610 10023 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } }
I20260810 11:37:34.725322 10017 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:34.727828 10025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } } }
I20260810 11:37:34.728142 10025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:34.729149 10033 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:34.728981 10026 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8ddea9e9505c49b68cb5426fe0e8050c. Latest consensus state: current_term: 1 leader_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8ddea9e9505c49b68cb5426fe0e8050c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39731 } } }
I20260810 11:37:34.729627 10026 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:34.738552 10033 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:34.753216 10033 catalog_manager.cc:1347] Generated new cluster ID: 233c2f87d0c949f3bd3733d0637e26c3
I20260810 11:37:34.753345 10033 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:34.767895 10033 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:34.769984 10033 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:34.792510 10033 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: Generated new TSK 0
I20260810 11:37:34.793532 10033 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:34.867929 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:35.234660 10052 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:35.235188 10052 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:35.235399 10052 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:35.244410 10052 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:35.244828 10052 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:35.245266 10052 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:35.256115 10052 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:35.257367 10052 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:35.258662 10052 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:35.273850 10058 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:35.273900 10060 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:35.273803 10062 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:35.275040 10052 server_base.cc:1034] running on GCE node
I20260810 11:37:35.275660 10052 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:35.276574 10052 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:35.277782 10052 hybrid_clock.cc:648] HybridClock initialized: now 1786361855277747 us; error 34 us; skew 500 ppm
I20260810 11:37:35.278071 10052 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:35 dist-test-slave-drl0 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:35.281343 10052 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:35.283577 10052 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:35.283730 10052 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:35.284092 10052 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:35.287333 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "f3197a15805c436a90c2dbbb2b8ff379"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "551a645fca203cb44af6282866cfbafb"
server_key_iv: "6f6a69ab133dfdbd3e868f1fb65c50a9"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.288198 10052 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "f3197a15805c436a90c2dbbb2b8ff379"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "551a645fca203cb44af6282866cfbafb"
server_key_iv: "6f6a69ab133dfdbd3e868f1fb65c50a9"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.294078 10052 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:35.298380 10073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.300443 10052 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:35.300735 10052 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "f3197a15805c436a90c2dbbb2b8ff379"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "551a645fca203cb44af6282866cfbafb"
server_key_iv: "6f6a69ab133dfdbd3e868f1fb65c50a9"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.301147 10052 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:35.321532 10052 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:35.327409 10052 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:35.327721 10052 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:35.328749 10052 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:35.330251 10052 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:35.330399 10052 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.330543 10052 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:35.330636 10052 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.378472 10052 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43153
I20260810 11:37:35.378511 10226 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43153 every 8 connection(s)
I20260810 11:37:35.379493 10052 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:35.388119 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10052
I20260810 11:37:35.388384 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:35.389035 4035 external_mini_cluster.cc:1428] Setting key 7f304e75e00a169e60dc02024ce590d1
Aug 10 11:37:35 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:35.403489 10230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:35.383197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46471 (local address 127.3.240.254:39731)
0810 11:37:35.383677 (+ 480us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:35.383682 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:35.384863 (+ 1181us) server_negotiation.cc:407] Connection header received
0810 11:37:35.387736 (+ 2873us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:35.387744 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:35.387873 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:35.388035 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:35.389901 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.390706 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.392574 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.392996 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.393141 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:35.396897 (+ 3756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:35.396902 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:35.396906 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:35.397017 (+ 111us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:35.399810 (+ 2793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:35.400607 (+ 797us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:35.400615 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:35.400622 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:35.400757 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:35.401500 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:35.401507 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:35.401515 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:35.401941 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:35.402126 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:35.402862 (+ 736us) server_negotiation.cc:299] Negotiation successful
0810 11:37:35.403128 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":262,"threads_started":1}
I20260810 11:37:35.404520 10231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:35.383862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39731 (local address 127.3.240.193:46471)
0810 11:37:35.384661 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:35.384714 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:35.387215 (+ 2501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:35.388334 (+ 1119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:35.388348 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:35.389003 (+ 655us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:35.389695 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:35.389719 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.390895 (+ 1176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.390907 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:35.392400 (+ 1493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:35.392414 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.393234 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.393242 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:35.393475 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:35.394178 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:35.394202 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:35.396664 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:35.399961 (+ 3297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:35.399970 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:35.399986 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:35.400409 (+ 423us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:35.401036 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:35.401044 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:35.401049 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:35.401340 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:35.402105 (+ 765us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:35.402113 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:35.402449 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 11:37:35.402717 (+ 268us) client_negotiation.cc:240] Negotiation successful
0810 11:37:35.403262 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":336,"threads_started":1}
I20260810 11:37:35.406338 10227 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39731
I20260810 11:37:35.406682 10227 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:35.407538 10227 heartbeater.cc:511] Master 127.3.240.254:39731 requested a full tablet report, sending...
I20260810 11:37:35.410220 9971 ts_manager.cc:194] Registered new tserver with Master: f3197a15805c436a90c2dbbb2b8ff379 (127.3.240.193:43153)
I20260810 11:37:35.412532 9971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:46471
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:35.450445 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:35.803407 10234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:35.803678 10234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:35.803776 10234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:35.811986 10234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:35.812093 10234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:35.812206 10234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:35.823186 10234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:35.824533 10234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:35.826030 10234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:35.845810 10243 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:35.845867 10239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:35.845927 10240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:35.847435 10234 server_base.cc:1034] running on GCE node
I20260810 11:37:35.848199 10234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:35.849249 10234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:35.850540 10234 hybrid_clock.cc:648] HybridClock initialized: now 1786361855850495 us; error 34 us; skew 500 ppm
I20260810 11:37:35.850904 10234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:35 dist-test-slave-drl0 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:35.859455 10234 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:35.860910 10234 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:35.861038 10234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:35.861330 10234 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:35.864378 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "84250dcaa0eb449abc1bb7975db12d7d"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "105936af629a0f63cdc0c54d711a9a11"
server_key_iv: "88d02d87daf05b88b721862b9fa8f2d1"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.865227 10234 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "84250dcaa0eb449abc1bb7975db12d7d"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "105936af629a0f63cdc0c54d711a9a11"
server_key_iv: "88d02d87daf05b88b721862b9fa8f2d1"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.870368 10234 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:35.874480 10251 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.876071 10234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:35.876269 10234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "84250dcaa0eb449abc1bb7975db12d7d"
format_stamp: "Formatted at 2026-08-10 11:37:35 on dist-test-slave-drl0"
server_key: "105936af629a0f63cdc0c54d711a9a11"
server_key_iv: "88d02d87daf05b88b721862b9fa8f2d1"
server_key_version: "encryptionkey@0"
I20260810 11:37:35.876505 10234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:35.891016 10234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:35.895620 10234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:35.895916 10234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:35.897119 10234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:35.898525 10234 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:35.898648 10234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.898789 10234 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:35.898881 10234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:35.939534 10234 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:44033
I20260810 11:37:35.939625 10415 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:44033 every 8 connection(s)
I20260810 11:37:35.940752 10234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:35.947073 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10234
I20260810 11:37:35.947281 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:35.947826 4035 external_mini_cluster.cc:1428] Setting key 3a731c8548b02549e7eaef675b30b03b
Aug 10 11:37:35 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361855, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:35.968609 10419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:35.944508 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:57577 (local address 127.3.240.254:39731)
0810 11:37:35.945030 (+ 522us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:35.945037 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:35.946473 (+ 1436us) server_negotiation.cc:407] Connection header received
0810 11:37:35.949041 (+ 2568us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:35.949046 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:35.949134 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:35.949387 (+ 253us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:35.951447 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.952140 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.954617 (+ 2477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.955291 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.955396 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:35.960330 (+ 4934us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:35.960335 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:35.960344 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:35.960376 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:35.964716 (+ 4340us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:35.965993 (+ 1277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:35.966001 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:35.966010 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:35.966170 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:35.966659 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:35.966663 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:35.966665 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:35.966989 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:35.967154 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:35.967869 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 11:37:35.968196 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":3,"server-negotiator.queue_time_us":395,"thread_start_us":249,"threads_started":1}
I20260810 11:37:35.969635 10420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:35.945531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39731 (local address 127.3.240.194:57577)
0810 11:37:35.946287 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:35.946329 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:35.948682 (+ 2353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:35.949602 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:35.949610 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:35.950304 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:35.951160 (+ 856us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:35.951188 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.952313 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.952321 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:35.954449 (+ 2128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:35.954467 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:35.955536 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:35.955545 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:35.955880 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:35.957007 (+ 1127us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:35.957146 (+ 139us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:35.960090 (+ 2944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:35.964992 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:35.965004 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:35.965024 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:35.965818 (+ 794us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:35.966324 (+ 506us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:35.966328 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:35.966330 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:35.966524 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:35.967134 (+ 610us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:35.967142 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:35.967480 (+ 338us) client_negotiation.cc:769] Sending connection context
0810 11:37:35.967754 (+ 274us) client_negotiation.cc:240] Negotiation successful
0810 11:37:35.968446 (+ 692us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":517,"thread_start_us":291,"threads_started":1}
I20260810 11:37:35.971252 10416 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39731
I20260810 11:37:35.971621 10416 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:35.972457 10416 heartbeater.cc:511] Master 127.3.240.254:39731 requested a full tablet report, sending...
I20260810 11:37:35.974182 9971 ts_manager.cc:194] Registered new tserver with Master: 84250dcaa0eb449abc1bb7975db12d7d (127.3.240.194:44033)
I20260810 11:37:35.975095 9971 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:57577
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:36.014051 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41065
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:36.386271 10425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:36.386642 10425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:36.386756 10425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:36.394681 10425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:36.394797 10425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:36.394917 10425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:36.407147 10425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41065
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39731
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:36.408458 10425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:36.409981 10425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260810 11:37:36.416649 10227 heartbeater.cc:503] Master 127.3.240.254:39731 was elected leader, sending a full tablet report...
W20260810 11:37:36.423900 10432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:36.424026 10433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:36.424247 10439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:36.425082 10425 server_base.cc:1034] running on GCE node
I20260810 11:37:36.425777 10425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:36.426893 10425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:36.428192 10425 hybrid_clock.cc:648] HybridClock initialized: now 1786361856428128 us; error 47 us; skew 500 ppm
I20260810 11:37:36.428542 10425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:36 dist-test-slave-drl0 krb5kdc[9905](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:36.431938 10425 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:36.433115 10425 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:36.433207 10425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:36.433461 10425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:36.436681 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "268d3bcec40049279177a539b1e0b930"
format_stamp: "Formatted at 2026-08-10 11:37:36 on dist-test-slave-drl0"
server_key: "72ccab9a52b20ce9339691d90ffaec65"
server_key_iv: "090d6c24ccd869c5852ad603fd575eba"
server_key_version: "encryptionkey@0"
I20260810 11:37:36.437491 10425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "268d3bcec40049279177a539b1e0b930"
format_stamp: "Formatted at 2026-08-10 11:37:36 on dist-test-slave-drl0"
server_key: "72ccab9a52b20ce9339691d90ffaec65"
server_key_iv: "090d6c24ccd869c5852ad603fd575eba"
server_key_version: "encryptionkey@0"
I20260810 11:37:36.442595 10425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:36.446748 10447 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:36.448508 10425 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:36.448642 10425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "268d3bcec40049279177a539b1e0b930"
format_stamp: "Formatted at 2026-08-10 11:37:36 on dist-test-slave-drl0"
server_key: "72ccab9a52b20ce9339691d90ffaec65"
server_key_iv: "090d6c24ccd869c5852ad603fd575eba"
server_key_version: "encryptionkey@0"
I20260810 11:37:36.448823 10425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:36.474638 10425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:36.479149 10425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:36.479398 10425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:36.480517 10425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:36.482048 10425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:36.482141 10425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:36.482215 10425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:36.482271 10425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:36.521715 10425 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:36481
I20260810 11:37:36.521816 10605 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:36481 every 8 connection(s)
I20260810 11:37:36.522627 10425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:36.531810 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10425
I20260810 11:37:36.532027 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:36.532562 4035 external_mini_cluster.cc:1428] Setting key 58e681b0789826c319bcbbf325d0c64f
Aug 10 11:37:36 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361856, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:36.546535 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.526131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:33511 (local address 127.3.240.254:39731)
0810 11:37:36.526630 (+ 499us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.526635 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.528096 (+ 1461us) server_negotiation.cc:407] Connection header received
0810 11:37:36.530287 (+ 2191us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.530292 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.530384 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.530515 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:36.532516 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.533539 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.535427 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.536112 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.536323 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.539858 (+ 3535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:36.539863 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:36.539866 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:36.539896 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:36.543010 (+ 3114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:36.544118 (+ 1108us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:36.544123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:36.544127 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:36.544322 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:36.544793 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:36.544796 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:36.544798 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:36.545077 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:36.545231 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.546016 (+ 785us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.546188 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":287,"threads_started":1}
I20260810 11:37:36.547633 10609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.526684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39731 (local address 127.3.240.195:33511)
0810 11:37:36.527849 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.527906 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.529972 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.530869 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.530879 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.531589 (+ 710us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:36.532290 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.532310 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.533717 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.533722 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.535267 (+ 1545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.535281 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.536478 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.536487 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.536700 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.537411 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:36.537434 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:36.539632 (+ 2198us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:36.543197 (+ 3565us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:36.543209 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:36.543225 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:36.543953 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:36.544458 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:36.544462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:36.544464 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:36.544648 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:36.545239 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:36.545246 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:36.545591 (+ 345us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.545827 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.546217 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":818,"thread_start_us":572,"threads_started":1}
I20260810 11:37:36.549244 10606 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39731
I20260810 11:37:36.549595 10606 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:36.550282 10606 heartbeater.cc:511] Master 127.3.240.254:39731 requested a full tablet report, sending...
I20260810 11:37:36.552397 9970 ts_manager.cc:194] Registered new tserver with Master: 268d3bcec40049279177a539b1e0b930 (127.3.240.195:36481)
I20260810 11:37:36.553375 9970 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:33511
I20260810 11:37:36.561966 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:37:36.583149 10608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.568012 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.3.240.254:39731)
0810 11:37:36.568283 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.568291 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.568938 (+ 647us) server_negotiation.cc:407] Connection header received
0810 11:37:36.569244 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.569257 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.569332 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.569620 (+ 288us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:36.571191 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.572044 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.573613 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.574039 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.574174 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.577140 (+ 2966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:36.577145 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:36.577149 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:36.577180 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:36.579873 (+ 2693us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:36.580559 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:36.580569 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:36.580571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:36.580689 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:36.581227 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:36.581233 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:36.581236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:36.581867 (+ 631us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:36.582158 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.582548 (+ 390us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.582729 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 11:37:36.587450 9970 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:36.595410 9970 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:36.623389 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.611052 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57746 (local address 127.3.240.195:36481)
0810 11:37:36.611699 (+ 647us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.611711 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.614180 (+ 2469us) server_negotiation.cc:407] Connection header received
0810 11:37:36.614503 (+ 323us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.614511 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.614731 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.614899 (+ 168us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:36.615774 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.617406 (+ 1632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.620464 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.621968 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.622118 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.622378 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.622789 (+ 411us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.622993 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":312,"threads_started":1}
I20260810 11:37:36.623472 10627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.610362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36481 (local address 127.0.0.1:57746)
0810 11:37:36.614055 (+ 3693us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.614076 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.614283 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.615124 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.615128 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.615155 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:36.615534 (+ 379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.615543 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.617582 (+ 2039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.617586 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.619841 (+ 2255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.619857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.622214 (+ 2357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.622219 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.622406 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.622447 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.622730 (+ 283us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.622993 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3532,"mutex_wait_us":3,"thread_start_us":3544,"threads_started":1}
I20260810 11:37:36.623922 10621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.610114 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43153 (local address 127.0.0.1:54360)
0810 11:37:36.611012 (+ 898us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.611066 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.611294 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.612692 (+ 1398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.612699 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.612740 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:36.613259 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.613281 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.615728 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.615742 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.618209 (+ 2467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.618224 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.622703 (+ 4479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.622712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.623068 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.623240 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.623343 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.623493 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":192,"thread_start_us":418,"threads_started":1}
I20260810 11:37:36.624141 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.612956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38230 (local address 127.3.240.194:44033)
0810 11:37:36.613503 (+ 547us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.613515 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.614363 (+ 848us) server_negotiation.cc:407] Connection header received
0810 11:37:36.614752 (+ 389us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.614759 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.615002 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.615147 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:36.616248 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.618505 (+ 2257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.620702 (+ 2197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.622087 (+ 1385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.622187 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.622347 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.623601 (+ 1254us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.623772 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":275,"threads_started":1}
I20260810 11:37:36.624733 10622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.612953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44033 (local address 127.0.0.1:38230)
0810 11:37:36.614255 (+ 1302us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.614272 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.614555 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.615399 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.615403 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.615447 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:36.616043 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.616060 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.618732 (+ 2672us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.618738 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.620502 (+ 1764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.620519 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.622703 (+ 2184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.622712 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.623070 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.623239 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.623378 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.623499 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1164,"mutex_wait_us":130,"thread_start_us":745,"threads_started":1}
I20260810 11:37:36.625067 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.610648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54360 (local address 127.3.240.193:43153)
0810 11:37:36.612000 (+ 1352us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.612008 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.612036 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:37:36.612133 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.612142 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.612407 (+ 265us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.613520 (+ 1113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:36.613800 (+ 280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.615539 (+ 1739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.618396 (+ 2857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.620185 (+ 1789us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.620477 (+ 292us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.620889 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.624487 (+ 3598us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.624687 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1188,"thread_start_us":1012,"threads_started":1}
I20260810 11:37:36.628110 10325 tablet_service.cc:1467] Processing CreateTablet for tablet 532bda768d7e405c87429503ee946959 (DEFAULT_TABLE table=test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:36.628320 10518 tablet_service.cc:1467] Processing CreateTablet for tablet 532bda768d7e405c87429503ee946959 (DEFAULT_TABLE table=test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:36.628715 10133 tablet_service.cc:1467] Processing CreateTablet for tablet 532bda768d7e405c87429503ee946959 (DEFAULT_TABLE table=test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:36.629791 10325 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532bda768d7e405c87429503ee946959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:36.629859 10518 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532bda768d7e405c87429503ee946959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:36.630108 10133 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 532bda768d7e405c87429503ee946959. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:36.644358 10628 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Bootstrap starting.
I20260810 11:37:36.647287 10629 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Bootstrap starting.
I20260810 11:37:36.648725 10628 tablet_bootstrap.cc:654] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:36.649997 10630 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Bootstrap starting.
I20260810 11:37:36.651206 10628 log.cc:824] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:36.651779 10629 tablet_bootstrap.cc:654] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:36.652997 10629 log.cc:824] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:36.653335 10630 tablet_bootstrap.cc:654] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:36.654881 10630 log.cc:824] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:36.655077 10628 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: No bootstrap required, opened a new log
I20260810 11:37:36.655407 10628 ts_tablet_manager.cc:1397] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 11:37:36.658506 10629 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: No bootstrap required, opened a new log
I20260810 11:37:36.658506 10630 tablet_bootstrap.cc:492] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: No bootstrap required, opened a new log
I20260810 11:37:36.658879 10630 ts_tablet_manager.cc:1397] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 11:37:36.658867 10629 ts_tablet_manager.cc:1397] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.006s
I20260810 11:37:36.663632 10628 raft_consensus.cc:348] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.664026 10628 raft_consensus.cc:374] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:36.664132 10628 raft_consensus.cc:729] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3197a15805c436a90c2dbbb2b8ff379, State: Initialized, Role: FOLLOWER
I20260810 11:37:36.664780 10628 consensus_queue.cc:260] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.666580 10628 ts_tablet_manager.cc:1428] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 11:37:36.670233 10630 raft_consensus.cc:348] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.670233 10629 raft_consensus.cc:348] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.670682 10630 raft_consensus.cc:374] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:36.670681 10629 raft_consensus.cc:374] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:36.671072 10629 raft_consensus.cc:729] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84250dcaa0eb449abc1bb7975db12d7d, State: Initialized, Role: FOLLOWER
I20260810 11:37:36.671077 10630 raft_consensus.cc:729] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 268d3bcec40049279177a539b1e0b930, State: Initialized, Role: FOLLOWER
I20260810 11:37:36.671711 10629 consensus_queue.cc:260] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.671711 10630 consensus_queue.cc:260] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.672931 10416 heartbeater.cc:503] Master 127.3.240.254:39731 was elected leader, sending a full tablet report...
I20260810 11:37:36.673807 10606 heartbeater.cc:503] Master 127.3.240.254:39731 was elected leader, sending a full tablet report...
I20260810 11:37:36.674921 10629 ts_tablet_manager.cc:1428] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Time spent starting tablet: real 0.016s user 0.004s sys 0.009s
I20260810 11:37:36.674958 10630 ts_tablet_manager.cc:1428] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Time spent starting tablet: real 0.016s user 0.012s sys 0.002s
W20260810 11:37:36.693708 10417 tablet.cc:2369] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:36.774418 10607 tablet.cc:2369] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:36.883496 10228 tablet.cc:2369] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:36.923413 10638 raft_consensus.cc:482] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:36.923841 10638 raft_consensus.cc:504] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.925716 10638 leader_election.cc:288] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f3197a15805c436a90c2dbbb2b8ff379 (127.3.240.193:43153), 268d3bcec40049279177a539b1e0b930 (127.3.240.195:36481)
I20260810 11:37:36.935956 10643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.926064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36481 (local address 127.3.240.194:50515)
0810 11:37:36.927094 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.927116 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.927291 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.927634 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.927639 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.927662 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:36.928782 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.928825 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.930567 (+ 1742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.930575 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.933256 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.933274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.935230 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.935236 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.935402 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.935425 (+ 23us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.935504 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.935596 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":883,"mutex_wait_us":3,"thread_start_us":795,"threads_started":1}
I20260810 11:37:36.935989 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.926412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:50515 (local address 127.3.240.195:36481)
0810 11:37:36.926659 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.926667 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.927217 (+ 550us) server_negotiation.cc:407] Connection header received
0810 11:37:36.927447 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.927451 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.927512 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.927616 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:36.929046 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.930370 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.933523 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.934986 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.935171 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.935253 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.935583 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.935716 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 11:37:36.937101 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.926156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:43153 (local address 127.3.240.194:53783)
0810 11:37:36.927161 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:36.927176 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:36.927338 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:36.927767 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:36.927771 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:36.927797 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:36.928789 (+ 992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.928807 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.930748 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.930759 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.933667 (+ 2908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:36.933689 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.936253 (+ 2564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.936259 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:36.936447 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.936483 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:37:36.936584 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 11:37:36.936733 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":911,"mutex_wait_us":5,"thread_start_us":421,"threads_started":1}
I20260810 11:37:36.937330 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.926124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53783 (local address 127.3.240.193:43153)
0810 11:37:36.926357 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.926366 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.927269 (+ 903us) server_negotiation.cc:407] Connection header received
0810 11:37:36.927484 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.927488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.927546 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.927648 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:36.929051 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.930552 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.933915 (+ 3363us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.934989 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.935186 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:36.935277 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:36.936739 (+ 1462us) server_negotiation.cc:299] Negotiation successful
0810 11:37:36.936949 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 11:37:36.937297 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532bda768d7e405c87429503ee946959" candidate_uuid: "84250dcaa0eb449abc1bb7975db12d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268d3bcec40049279177a539b1e0b930" is_pre_election: true
I20260810 11:37:36.937849 10546 raft_consensus.cc:2454] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84250dcaa0eb449abc1bb7975db12d7d in term 0.
I20260810 11:37:36.938521 10164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532bda768d7e405c87429503ee946959" candidate_uuid: "84250dcaa0eb449abc1bb7975db12d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3197a15805c436a90c2dbbb2b8ff379" is_pre_election: true
I20260810 11:37:36.938730 10255 leader_election.cc:302] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 268d3bcec40049279177a539b1e0b930, 84250dcaa0eb449abc1bb7975db12d7d; no voters:
I20260810 11:37:36.939041 10164 raft_consensus.cc:2454] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 84250dcaa0eb449abc1bb7975db12d7d in term 0.
I20260810 11:37:36.939251 10638 raft_consensus.cc:2781] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:36.939399 10638 raft_consensus.cc:482] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:36.939507 10638 raft_consensus.cc:3037] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:36.942018 10638 raft_consensus.cc:504] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.942651 10638 leader_election.cc:288] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [CANDIDATE]: Term 1 election: Requested vote from peers f3197a15805c436a90c2dbbb2b8ff379 (127.3.240.193:43153), 268d3bcec40049279177a539b1e0b930 (127.3.240.195:36481)
I20260810 11:37:36.943025 10164 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532bda768d7e405c87429503ee946959" candidate_uuid: "84250dcaa0eb449abc1bb7975db12d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f3197a15805c436a90c2dbbb2b8ff379"
I20260810 11:37:36.943238 10164 raft_consensus.cc:3037] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:36.943281 10546 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "532bda768d7e405c87429503ee946959" candidate_uuid: "84250dcaa0eb449abc1bb7975db12d7d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "268d3bcec40049279177a539b1e0b930"
I20260810 11:37:36.943482 10546 raft_consensus.cc:3037] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:36.946436 10164 raft_consensus.cc:2454] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84250dcaa0eb449abc1bb7975db12d7d in term 1.
I20260810 11:37:36.947060 10254 leader_election.cc:302] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84250dcaa0eb449abc1bb7975db12d7d, f3197a15805c436a90c2dbbb2b8ff379; no voters:
I20260810 11:37:36.947134 10546 raft_consensus.cc:2454] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 84250dcaa0eb449abc1bb7975db12d7d in term 1.
I20260810 11:37:36.947450 10638 raft_consensus.cc:2781] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:36.947934 10638 raft_consensus.cc:686] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 1 LEADER]: Becoming Leader. State: Replica: 84250dcaa0eb449abc1bb7975db12d7d, State: Running, Role: LEADER
I20260810 11:37:36.948524 10638 consensus_queue.cc:237] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } }
I20260810 11:37:36.953384 9970 catalog_manager.cc:5495] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d reported cstate change: term changed from 0 to 1, leader changed from <none> to 84250dcaa0eb449abc1bb7975db12d7d (127.3.240.194). New cstate: current_term: 1 leader_uuid: "84250dcaa0eb449abc1bb7975db12d7d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "84250dcaa0eb449abc1bb7975db12d7d" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44033 } health_report { overall_health: HEALTHY } } }
I20260810 11:37:37.001303 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:36.995487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38244 (local address 127.3.240.194:44033)
0810 11:37:36.995930 (+ 443us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:36.995938 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:36.995964 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:37:36.996081 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:36.996088 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:36.996185 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:36.996326 (+ 141us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:37:36.996749 (+ 423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.997470 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.998905 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:36.999546 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:36.999660 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:37.000175 (+ 515us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:37:37.000687 (+ 512us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:37:37.000803 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:37.001000 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 11:37:37.001069 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 11:37:37.009783 10546 raft_consensus.cc:1261] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 1 FOLLOWER]: Refusing update from remote peer 84250dcaa0eb449abc1bb7975db12d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:37.009827 10164 raft_consensus.cc:1261] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 1 FOLLOWER]: Refusing update from remote peer 84250dcaa0eb449abc1bb7975db12d7d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:37.011031 10638 consensus_queue.cc:1035] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "f3197a15805c436a90c2dbbb2b8ff379" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 43153 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:37.011355 10645 consensus_queue.cc:1035] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [LEADER]: Connected to new peer: Peer: permanent_uuid: "268d3bcec40049279177a539b1e0b930" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:37.023617 10648 mvcc.cc:205] Tried to move back new op lower bound from 7316938166302625792 to 7316938166065283072. Current Snapshot: MvccSnapshot[applied={T|T < 7316938166302625792}]
I20260810 11:37:37.025472 10651 mvcc.cc:205] Tried to move back new op lower bound from 7316938166302625792 to 7316938166065283072. Current Snapshot: MvccSnapshot[applied={T|T < 7316938166302625792}]
I20260810 11:37:37.038460 9970 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:37.038748 9970 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:37.043416 9970 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8ddea9e9505c49b68cb5426fe0e8050c: Sending DeleteTablet for 3 replicas of tablet 532bda768d7e405c87429503ee946959
I20260810 11:37:37.044902 10518 tablet_service.cc:1514] Processing DeleteTablet for tablet 532bda768d7e405c87429503ee946959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:57746
I20260810 11:37:37.044888 10133 tablet_service.cc:1514] Processing DeleteTablet for tablet 532bda768d7e405c87429503ee946959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:54360
I20260810 11:37:37.045245 10325 tablet_service.cc:1514] Processing DeleteTablet for tablet 532bda768d7e405c87429503ee946959 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:38230
I20260810 11:37:37.048043 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10052
I20260810 11:37:37.057572 10659 tablet_replica.cc:330] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: stopping tablet replica
I20260810 11:37:37.057926 10661 tablet_replica.cc:330] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: stopping tablet replica
I20260810 11:37:37.058585 10661 raft_consensus.cc:2229] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:37.058866 10659 raft_consensus.cc:2229] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:37.059417 10659 raft_consensus.cc:2258] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:37.059461 10661 raft_consensus.cc:2258] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:37.060400 10660 tablet_replica.cc:330] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: stopping tablet replica
I20260810 11:37:37.060856 10660 raft_consensus.cc:2229] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:37.061414 10660 raft_consensus.cc:2258] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:37.062270 10661 ts_tablet_manager.cc:1905] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:37.063791 10659 ts_tablet_manager.cc:1905] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:37.065217 10660 ts_tablet_manager.cc:1905] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:37.071440 10661 ts_tablet_manager.cc:1918] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:37.071642 10661 log.cc:1196] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/532bda768d7e405c87429503ee946959
I20260810 11:37:37.072230 10661 ts_tablet_manager.cc:1939] T 532bda768d7e405c87429503ee946959 P 84250dcaa0eb449abc1bb7975db12d7d: Deleting consensus metadata
I20260810 11:37:37.073659 10660 ts_tablet_manager.cc:1918] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:37.073935 10660 log.cc:1196] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/532bda768d7e405c87429503ee946959
I20260810 11:37:37.074415 9952 catalog_manager.cc:4841] TS 84250dcaa0eb449abc1bb7975db12d7d (127.3.240.194:44033): tablet 532bda768d7e405c87429503ee946959 (table test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]) successfully deleted
I20260810 11:37:37.074581 10659 ts_tablet_manager.cc:1918] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:37.074759 10659 log.cc:1196] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/532bda768d7e405c87429503ee946959
I20260810 11:37:37.074788 10660 ts_tablet_manager.cc:1939] T 532bda768d7e405c87429503ee946959 P f3197a15805c436a90c2dbbb2b8ff379: Deleting consensus metadata
I20260810 11:37:37.075309 10659 ts_tablet_manager.cc:1939] T 532bda768d7e405c87429503ee946959 P 268d3bcec40049279177a539b1e0b930: Deleting consensus metadata
Aug 10 11:37:37 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:37.077013 9949 catalog_manager.cc:4841] TS 268d3bcec40049279177a539b1e0b930 (127.3.240.195:36481): tablet 532bda768d7e405c87429503ee946959 (table test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]) successfully deleted
I20260810 11:37:37.077103 9944 catalog_manager.cc:4841] TS f3197a15805c436a90c2dbbb2b8ff379 (127.3.240.193:43153): tablet 532bda768d7e405c87429503ee946959 (table test-table [id=ec79ce72e8174bbd97e3ea657c3bcc75]) successfully deleted
I20260810 11:37:37.083153 10623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:37.049444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54366 (local address 127.3.240.193:43153)
0810 11:37:37.049663 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:37.049670 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:37.054223 (+ 4553us) server_negotiation.cc:407] Connection header received
0810 11:37:37.054554 (+ 331us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:37.054563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:37.054652 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:37.054767 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:37.059467 (+ 4700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.060731 (+ 1264us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.062624 (+ 1893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.063268 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.063385 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:37.076648 (+ 13263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:37.076663 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:37.076681 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:37.076742 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:37.080051 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.080994 (+ 943us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.081002 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.081005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.081147 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.081604 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.081608 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.081610 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.081973 (+ 363us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:37.082219 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:37.082631 (+ 412us) server_negotiation.cc:299] Negotiation successful
0810 11:37:37.082803 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 11:37:37.083917 10218 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:37.183696 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10052
I20260810 11:37:37.221906 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10234
Aug 10 11:37:37 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:37.240854 10626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:37.222743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38256 (local address 127.3.240.194:44033)
0810 11:37:37.223033 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:37.223038 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:37.223058 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:37:37.223263 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:37.223268 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:37.223349 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:37.223445 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:37.225094 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.225990 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.227893 (+ 1903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.228437 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.228553 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:37.234280 (+ 5727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:37.234302 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:37.234324 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:37.234374 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:37.237683 (+ 3309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.238447 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.238454 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.238457 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.238594 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.239031 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.239035 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.239037 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.239595 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:37.239997 (+ 402us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:37.240258 (+ 261us) server_negotiation.cc:299] Negotiation successful
0810 11:37:37.240501 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:37:37.241606 10410 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:37.359664 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10234
I20260810 11:37:37.386145 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10425
Aug 10 11:37:37 dist-test-slave-drl0 krb5kdc[9905](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361854, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:37.404307 10624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:37.387041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57748 (local address 127.3.240.195:36481)
0810 11:37:37.387234 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:37.387238 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:37.387256 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:37.387369 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:37.387373 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:37.387435 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:37.387529 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:37.388801 (+ 1272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.389514 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.391500 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:37.392036 (+ 536us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:37.392143 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:37.397385 (+ 5242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:37.397398 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:37.397415 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:37.397522 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:37.401154 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.401962 (+ 808us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.401974 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.401978 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.402177 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:37.402669 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:37.402675 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:37.402679 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:37.403205 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:37.403454 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:37.403771 (+ 317us) server_negotiation.cc:299] Negotiation successful
0810 11:37:37.403937 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 11:37:37.405056 10600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:37.498888 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10425
I20260810 11:37:37.522717 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 9922
I20260810 11:37:37.523998 10012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:37.643646 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 9922
2026-08-10T11:37:37Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5053 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-10T11:37:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:37Z Disabled control of system clock
I20260810 11:37:37.737427 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39763
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39763
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 11:37:38.108981 10671 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:38.109313 10671 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:38.109478 10671 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:38.116782 10671 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:38.116888 10671 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:38.116927 10671 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:38.117017 10671 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:38.117087 10671 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:38.127377 10671 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39763
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39763
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:38.128753 10671 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:38.131153 10671 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:38.144444 10676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:38.145602 10671 server_base.cc:1034] running on GCE node
W20260810 11:37:38.148537 10681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:38.148813 10679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:38.149694 10671 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:38.151470 10671 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:38.152712 10671 hybrid_clock.cc:648] HybridClock initialized: now 1786361858152668 us; error 34 us; skew 500 ppm
I20260810 11:37:38.153853 10671 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:38.156868 10671 webserver.cc:460] Webserver started at http://127.3.240.254:34363/ using document root <none> and password file <none>
I20260810 11:37:38.157614 10671 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:38.157727 10671 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:38.158213 10671 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:38.161458 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "f06959bb5e1a428289b0ea0293f978da"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "c657bdd6f250cc8a657d835564e90885"
server_key_iv: "0d3cb6f43e912501839514efbf57e4c6"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.162281 10671 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f06959bb5e1a428289b0ea0293f978da"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "c657bdd6f250cc8a657d835564e90885"
server_key_iv: "0d3cb6f43e912501839514efbf57e4c6"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.168469 10671 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.006s
I20260810 11:37:38.173048 10689 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:38.174502 10671 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:37:38.174707 10671 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "f06959bb5e1a428289b0ea0293f978da"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "c657bdd6f250cc8a657d835564e90885"
server_key_iv: "0d3cb6f43e912501839514efbf57e4c6"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.174983 10671 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:38.181056 10671 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:38.181828 10671 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:38.182140 10671 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:38.209832 10671 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39763
I20260810 11:37:38.209873 10754 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39763 every 8 connection(s)
I20260810 11:37:38.212037 10671 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:38.216105 10755 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:38.221923 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10671
I20260810 11:37:38.222127 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:38.222833 4035 external_mini_cluster.cc:1428] Setting key ec7d97fcd87ae6a04f57a97f4ec322af
I20260810 11:37:38.227118 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: Bootstrap starting.
I20260810 11:37:38.230954 10755 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:38.232103 10755 log.cc:824] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:38.232358 10759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:38.225044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.3.240.254:39763)
0810 11:37:38.226124 (+ 1080us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:38.226136 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:38.226179 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:37:38.227914 (+ 1735us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:38.227935 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:38.228310 (+ 375us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:38.228641 (+ 331us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:38.229828 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:38.229836 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:38.229933 (+ 97us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:38.229958 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:38.230034 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:38.230655 (+ 621us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:38.230851 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:38.231157 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 11:37:38.231450 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":372,"threads_started":1}
I20260810 11:37:38.235076 10755 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: No bootstrap required, opened a new log
I20260810 11:37:38.242807 10755 raft_consensus.cc:348] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } }
I20260810 11:37:38.243216 10755 raft_consensus.cc:374] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:38.243428 10755 raft_consensus.cc:729] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f06959bb5e1a428289b0ea0293f978da, State: Initialized, Role: FOLLOWER
I20260810 11:37:38.244442 10755 consensus_queue.cc:260] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } }
I20260810 11:37:38.244690 10755 raft_consensus.cc:388] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:38.244833 10755 raft_consensus.cc:482] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:38.245015 10755 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:38.247570 10755 raft_consensus.cc:504] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } }
I20260810 11:37:38.248170 10755 leader_election.cc:302] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f06959bb5e1a428289b0ea0293f978da; no voters:
I20260810 11:37:38.248932 10755 leader_election.cc:288] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:38.249155 10761 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:38.250515 10761 raft_consensus.cc:686] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [term 1 LEADER]: Becoming Leader. State: Replica: f06959bb5e1a428289b0ea0293f978da, State: Running, Role: LEADER
I20260810 11:37:38.251094 10761 consensus_queue.cc:237] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } }
I20260810 11:37:38.251672 10755 sys_catalog.cc:564] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:38.254207 10764 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [sys.catalog]: SysCatalogTable state changed. Reason: New leader f06959bb5e1a428289b0ea0293f978da. Latest consensus state: current_term: 1 leader_uuid: "f06959bb5e1a428289b0ea0293f978da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } } }
I20260810 11:37:38.254213 10763 sys_catalog.cc:455] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f06959bb5e1a428289b0ea0293f978da" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f06959bb5e1a428289b0ea0293f978da" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39763 } } }
I20260810 11:37:38.254635 10764 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:38.254637 10763 sys_catalog.cc:458] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:38.255560 10768 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:38.263664 10768 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:38.287635 10768 catalog_manager.cc:1347] Generated new cluster ID: 7a2360aa297e449ab5b18743446b3167
I20260810 11:37:38.287763 10768 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:38.311316 10768 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:38.312062 10768 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:38.319890 10768 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: Generated new TSK 0
I20260810 11:37:38.320849 10768 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:37:38.336082 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--builtin_ntp_servers=127.3.240.212:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260810 11:37:38.699730 10785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:38.700014 10785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:38.700183 10785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:38.707876 10785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:38.707978 10785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:38.708199 10785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:38.717435 10785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:38.718886 10785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:38.720769 10785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:38.735962 10792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:38.736140 10797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:38.736428 10793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:38.737064 10785 server_base.cc:1034] running on GCE node
I20260810 11:37:38.737654 10785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:38.738549 10785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:38.739771 10785 hybrid_clock.cc:648] HybridClock initialized: now 1786361858739721 us; error 45 us; skew 500 ppm
I20260810 11:37:38.740151 10785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:38.743312 10785 webserver.cc:460] Webserver started at http://127.3.240.193:39635/ using document root <none> and password file <none>
I20260810 11:37:38.744467 10785 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:38.744637 10785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:38.744997 10785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:38.748097 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "d523ef6858854bb797839337ac31dd44"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "a65eb782ff1bbcb4a94a5f09e82e67e2"
server_key_iv: "03f2392190588e4eadab33e0ec5a2c9e"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.748919 10785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "d523ef6858854bb797839337ac31dd44"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "a65eb782ff1bbcb4a94a5f09e82e67e2"
server_key_iv: "03f2392190588e4eadab33e0ec5a2c9e"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.754504 10785 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 11:37:38.758553 10812 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:38.760174 10785 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 11:37:38.760375 10785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "d523ef6858854bb797839337ac31dd44"
format_stamp: "Formatted at 2026-08-10 11:37:38 on dist-test-slave-drl0"
server_key: "a65eb782ff1bbcb4a94a5f09e82e67e2"
server_key_iv: "03f2392190588e4eadab33e0ec5a2c9e"
server_key_version: "encryptionkey@0"
I20260810 11:37:38.760627 10785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:38.767158 10785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:38.767971 10785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:38.768353 10785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:38.769349 10785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:38.770794 10785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:38.770931 10785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:38.771121 10785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:38.771217 10785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:38.811726 10785 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45921
I20260810 11:37:38.811815 10962 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45921 every 8 connection(s)
I20260810 11:37:38.813103 10785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:38.817916 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10785
I20260810 11:37:38.818234 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:38.819128 4035 external_mini_cluster.cc:1428] Setting key 8c749da8d531969e83607523c2044dc8
I20260810 11:37:38.822369 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--builtin_ntp_servers=127.3.240.212:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 11:37:38.823597 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:38.815944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:32783 (local address 127.3.240.254:39763)
0810 11:37:38.816453 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:38.816461 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:38.817731 (+ 1270us) server_negotiation.cc:407] Connection header received
0810 11:37:38.820243 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:38.820249 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:38.820317 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:38.820432 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:38.822164 (+ 1732us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:38.822170 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:38.822176 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:38.822217 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:38.822269 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:38.822286 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:38.822569 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:38.823022 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 11:37:38.823312 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20260810 11:37:38.824529 10971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:38.816507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39763 (local address 127.3.240.193:32783)
0810 11:37:38.817516 (+ 1009us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:38.817568 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:38.819934 (+ 2366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:38.820717 (+ 783us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:38.820727 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:38.820836 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:38.821855 (+ 1019us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:38.821858 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:38.821902 (+ 44us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:38.821993 (+ 91us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:38.822631 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:38.822640 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:38.822647 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 11:37:38.823341 (+ 694us) client_negotiation.cc:240] Negotiation successful
0810 11:37:38.823604 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":290,"threads_started":1}
I20260810 11:37:38.826197 10966 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39763
I20260810 11:37:38.826527 10966 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:38.827190 10966 heartbeater.cc:511] Master 127.3.240.254:39763 requested a full tablet report, sending...
I20260810 11:37:38.829591 10713 ts_manager.cc:194] Registered new tserver with Master: d523ef6858854bb797839337ac31dd44 (127.3.240.193:45921)
W20260810 11:37:39.180418 10972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:39.180742 10972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:39.180899 10972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:39.188591 10972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:39.188743 10972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:39.188935 10972 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:39.199314 10972 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:39.201066 10972 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:39.204119 10972 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:39.219020 10979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:39.219020 10981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:39.223323 10988 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:39.225056 10972 server_base.cc:1034] running on GCE node
I20260810 11:37:39.226146 10972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:39.227635 10972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:39.228916 10972 hybrid_clock.cc:648] HybridClock initialized: now 1786361859228842 us; error 68 us; skew 500 ppm
I20260810 11:37:39.229295 10972 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:39.232074 10972 webserver.cc:460] Webserver started at http://127.3.240.194:38499/ using document root <none> and password file <none>
I20260810 11:37:39.232820 10972 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:39.232955 10972 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:39.233299 10972 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:39.236255 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "4796049399c54856a61309fcd74524f3"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "a716127446ad2b7fef0d1565e6516985"
server_key_iv: "72719b5c9387d326e452a98476450c9f"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.237068 10972 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "4796049399c54856a61309fcd74524f3"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "a716127446ad2b7fef0d1565e6516985"
server_key_iv: "72719b5c9387d326e452a98476450c9f"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.244140 10972 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260810 11:37:39.248992 10997 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:39.251173 10972 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:39.251427 10972 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "4796049399c54856a61309fcd74524f3"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "a716127446ad2b7fef0d1565e6516985"
server_key_iv: "72719b5c9387d326e452a98476450c9f"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.251734 10972 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:39.258787 10972 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:39.259665 10972 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:39.259959 10972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:39.261014 10972 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:39.262673 10972 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:39.262768 10972 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:39.262897 10972 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:39.263031 10972 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 11:37:39.303732 10972 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36649
I20260810 11:37:39.303828 11151 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36649 every 8 connection(s)
I20260810 11:37:39.305078 10972 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:39.312081 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 10972
I20260810 11:37:39.312252 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:39.312867 4035 external_mini_cluster.cc:1428] Setting key 8d3c385e6c870155c5273f4fcc7b43af
I20260810 11:37:39.315632 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.308699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53451 (local address 127.3.240.254:39763)
0810 11:37:39.308978 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.308982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.310675 (+ 1693us) server_negotiation.cc:407] Connection header received
0810 11:37:39.313039 (+ 2364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.313044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.313106 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.313205 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.314760 (+ 1555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.314766 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.314771 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.314798 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.314846 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.314881 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.315058 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.315284 (+ 226us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.315346 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 11:37:39.316256 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--builtin_ntp_servers=127.3.240.212:41267
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260810 11:37:39.316731 11155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.309258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39763 (local address 127.3.240.194:53451)
0810 11:37:39.310438 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.310490 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.312602 (+ 2112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.313522 (+ 920us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.313532 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.313628 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.314455 (+ 827us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.314458 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.314497 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.314587 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.315022 (+ 435us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.315033 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.315042 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.315329 (+ 287us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.315535 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":381,"threads_started":1}
I20260810 11:37:39.318252 11153 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39763
I20260810 11:37:39.318600 11153 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:39.319402 11153 heartbeater.cc:511] Master 127.3.240.254:39763 requested a full tablet report, sending...
I20260810 11:37:39.321676 10713 ts_manager.cc:194] Registered new tserver with Master: 4796049399c54856a61309fcd74524f3 (127.3.240.194:36649)
W20260810 11:37:39.677621 11157 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:39.677940 11157 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:39.678114 11157 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:39.685602 11157 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:39.685741 11157 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:39.685931 11157 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:39.695246 11157 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41267
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:39.696664 11157 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:39.699155 11157 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:39.714422 11165 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:39.714481 11166 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:39.714881 11168 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:39.715727 11157 server_base.cc:1034] running on GCE node
I20260810 11:37:39.716609 11157 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:39.717790 11157 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:39.719074 11157 hybrid_clock.cc:648] HybridClock initialized: now 1786361859719022 us; error 63 us; skew 500 ppm
I20260810 11:37:39.719420 11157 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:39.722622 11157 webserver.cc:460] Webserver started at http://127.3.240.195:44725/ using document root <none> and password file <none>
I20260810 11:37:39.723459 11157 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:39.723590 11157 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:39.723946 11157 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:39.727240 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "d910f08a358f40779af8f7e69f8ccd07"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "9bdab3f908f0ddb3aa38d9753d398a98"
server_key_iv: "f42cc6ffa8f10052aa587a4712ea97b1"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.728084 11157 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "d910f08a358f40779af8f7e69f8ccd07"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "9bdab3f908f0ddb3aa38d9753d398a98"
server_key_iv: "f42cc6ffa8f10052aa587a4712ea97b1"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.733811 11157 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:39.737999 11176 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:39.740145 11157 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:39.740356 11157 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "d910f08a358f40779af8f7e69f8ccd07"
format_stamp: "Formatted at 2026-08-10 11:37:39 on dist-test-slave-drl0"
server_key: "9bdab3f908f0ddb3aa38d9753d398a98"
server_key_iv: "f42cc6ffa8f10052aa587a4712ea97b1"
server_key_version: "encryptionkey@0"
I20260810 11:37:39.740609 11157 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:39.747829 11157 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:39.748660 11157 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:39.748955 11157 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:39.750027 11157 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:39.751564 11157 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:39.751703 11157 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:39.751784 11157 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:39.751900 11157 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:39.791867 11157 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41383
I20260810 11:37:39.791960 11338 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41383 every 8 connection(s)
I20260810 11:37:39.793217 11157 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:39.798242 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 11157
I20260810 11:37:39.798605 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:39.799270 4035 external_mini_cluster.cc:1428] Setting key b1f099d322daf7998012f35f1713a0b2
I20260810 11:37:39.803872 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.796801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:50807 (local address 127.3.240.254:39763)
0810 11:37:39.797121 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.797126 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.798784 (+ 1658us) server_negotiation.cc:407] Connection header received
0810 11:37:39.801039 (+ 2255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.801044 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.801134 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.801249 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.803048 (+ 1799us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.803053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.803056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.803075 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.803111 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.803125 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.803305 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.803577 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.803638 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260810 11:37:39.804864 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.797383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39763 (local address 127.3.240.195:50807)
0810 11:37:39.798584 (+ 1201us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.798624 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.800683 (+ 2059us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.801503 (+ 820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.801511 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.801599 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.802573 (+ 974us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.802576 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.802624 (+ 48us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.802740 (+ 116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.803275 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.803287 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.803296 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.803643 (+ 347us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.803889 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":397,"threads_started":1}
I20260810 11:37:39.806531 11339 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39763
I20260810 11:37:39.807137 11339 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:39.808192 11339 heartbeater.cc:511] Master 127.3.240.254:39763 requested a full tablet report, sending...
I20260810 11:37:39.809958 10713 ts_manager.cc:194] Registered new tserver with Master: d910f08a358f40779af8f7e69f8ccd07 (127.3.240.195:41383)
I20260810 11:37:39.816385 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:37:39.824151 10970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.821420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56706 (local address 127.3.240.254:39763)
0810 11:37:39.821666 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.821676 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.822068 (+ 392us) server_negotiation.cc:407] Connection header received
0810 11:37:39.822191 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.822195 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.822251 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.822340 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.823429 (+ 1089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.823433 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.823437 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.823452 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.823483 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.823494 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.823603 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.823795 (+ 192us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.823885 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 11:37:39.833436 10966 heartbeater.cc:503] Master 127.3.240.254:39763 was elected leader, sending a full tablet report...
I20260810 11:37:39.829286 10713 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:56706:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:39.837410 10713 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:39.857889 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.851497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37266 (local address 127.3.240.193:45921)
0810 11:37:39.853082 (+ 1585us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.853090 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.854098 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 11:37:39.854188 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.854196 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.854573 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.854759 (+ 186us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.856838 (+ 2079us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.856848 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.856862 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.856883 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.856944 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.856969 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.857133 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.857391 (+ 258us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.857490 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1363,"thread_start_us":1194,"threads_started":1}
I20260810 11:37:39.858422 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.851144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33532 (local address 127.3.240.194:36649)
0810 11:37:39.852569 (+ 1425us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.852580 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.854029 (+ 1449us) server_negotiation.cc:407] Connection header received
0810 11:37:39.854225 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.854233 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.854472 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.854599 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.857149 (+ 2550us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.857159 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.857176 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.857208 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.857308 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.857399 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.857668 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.857946 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.858063 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1262,"thread_start_us":1138,"threads_started":1}
I20260810 11:37:39.858476 11354 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.851920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37266)
0810 11:37:39.853826 (+ 1906us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.853898 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.854051 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.855319 (+ 1268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.855326 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.855410 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.856530 (+ 1120us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.856536 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.856568 (+ 32us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.856644 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.857207 (+ 563us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.857215 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.857220 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.857399 (+ 179us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.857484 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1717,"spinlock_wait_cycles":4224,"thread_start_us":1316,"threads_started":1}
I20260810 11:37:39.859519 11360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.851030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36649 (local address 127.0.0.1:33532)
0810 11:37:39.853826 (+ 2796us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.853898 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.854051 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.855698 (+ 1647us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.855705 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.855730 (+ 25us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.856834 (+ 1104us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.856837 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.856852 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.856892 (+ 40us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.857747 (+ 855us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.857751 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.857756 (+ 5us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.858853 (+ 1097us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.858922 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":2418,"threads_started":1}
I20260810 11:37:39.861267 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.855045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41383 (local address 127.0.0.1:45362)
0810 11:37:39.856493 (+ 1448us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.856517 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.856620 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.859047 (+ 2427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.859052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.859080 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.859984 (+ 904us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.859990 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.860009 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.860057 (+ 48us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.860583 (+ 526us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.860588 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.860594 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.860751 (+ 157us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.860801 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1303,"thread_start_us":1224,"threads_started":1}
I20260810 11:37:39.861299 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.854436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45362 (local address 127.3.240.195:41383)
0810 11:37:39.856496 (+ 2060us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.856506 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.856783 (+ 277us) server_negotiation.cc:407] Connection header received
0810 11:37:39.856966 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.856986 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.857308 (+ 322us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.857641 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.860300 (+ 2659us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.860310 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.860325 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.860351 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.860415 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.860439 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.860661 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.860740 (+ 79us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.860839 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1655,"thread_start_us":1460,"threads_started":1}
I20260810 11:37:39.862864 10880 tablet_service.cc:1467] Processing CreateTablet for tablet bbadd95a143c46fba621e108c6782a0a (DEFAULT_TABLE table=test-table [id=6e2313a722424357b4713fb0889e33a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:39.862893 11058 tablet_service.cc:1467] Processing CreateTablet for tablet bbadd95a143c46fba621e108c6782a0a (DEFAULT_TABLE table=test-table [id=6e2313a722424357b4713fb0889e33a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:39.864269 10880 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbadd95a143c46fba621e108c6782a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:39.864274 11058 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbadd95a143c46fba621e108c6782a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:39.875110 11247 tablet_service.cc:1467] Processing CreateTablet for tablet bbadd95a143c46fba621e108c6782a0a (DEFAULT_TABLE table=test-table [id=6e2313a722424357b4713fb0889e33a8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:39.876762 11247 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bbadd95a143c46fba621e108c6782a0a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:39.878756 11364 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: Bootstrap starting.
I20260810 11:37:39.881348 11365 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Bootstrap starting.
I20260810 11:37:39.882771 11364 tablet_bootstrap.cc:654] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:39.884116 11364 log.cc:824] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:39.885046 11365 tablet_bootstrap.cc:654] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:39.886291 11365 log.cc:824] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:39.887667 11364 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: No bootstrap required, opened a new log
I20260810 11:37:39.888019 11364 ts_tablet_manager.cc:1397] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260810 11:37:39.891209 11365 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: No bootstrap required, opened a new log
I20260810 11:37:39.891610 11365 ts_tablet_manager.cc:1397] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260810 11:37:39.892982 11370 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Bootstrap starting.
I20260810 11:37:39.897310 11370 tablet_bootstrap.cc:654] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:39.897567 11364 raft_consensus.cc:348] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.897954 11364 raft_consensus.cc:374] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:39.898080 11364 raft_consensus.cc:729] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d523ef6858854bb797839337ac31dd44, State: Initialized, Role: FOLLOWER
I20260810 11:37:39.898734 11364 consensus_queue.cc:260] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.899080 11370 log.cc:824] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:39.900394 11364 ts_tablet_manager.cc:1428] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: Time spent starting tablet: real 0.012s user 0.004s sys 0.007s
I20260810 11:37:39.904088 11370 tablet_bootstrap.cc:492] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: No bootstrap required, opened a new log
I20260810 11:37:39.904404 11370 ts_tablet_manager.cc:1397] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20260810 11:37:39.904659 11365 raft_consensus.cc:348] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.905124 11365 raft_consensus.cc:374] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:39.905251 11365 raft_consensus.cc:729] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4796049399c54856a61309fcd74524f3, State: Initialized, Role: FOLLOWER
I20260810 11:37:39.905827 11365 consensus_queue.cc:260] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.908143 11153 heartbeater.cc:503] Master 127.3.240.254:39763 was elected leader, sending a full tablet report...
I20260810 11:37:39.908704 11365 ts_tablet_manager.cc:1428] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Time spent starting tablet: real 0.017s user 0.012s sys 0.003s
I20260810 11:37:39.914778 11370 raft_consensus.cc:348] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.915225 11370 raft_consensus.cc:374] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:39.915345 11370 raft_consensus.cc:729] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d910f08a358f40779af8f7e69f8ccd07, State: Initialized, Role: FOLLOWER
I20260810 11:37:39.915879 11370 consensus_queue.cc:260] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.916889 11339 heartbeater.cc:503] Master 127.3.240.254:39763 was elected leader, sending a full tablet report...
I20260810 11:37:39.917309 11370 ts_tablet_manager.cc:1428] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Time spent starting tablet: real 0.013s user 0.009s sys 0.003s
I20260810 11:37:39.960331 11375 raft_consensus.cc:482] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:39.960772 11375 raft_consensus.cc:504] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.962538 11375 leader_election.cc:288] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4796049399c54856a61309fcd74524f3 (127.3.240.194:36649), d523ef6858854bb797839337ac31dd44 (127.3.240.193:45921)
I20260810 11:37:39.965574 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.963375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37669 (local address 127.3.240.194:36649)
0810 11:37:39.963561 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.963565 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.963581 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:37:39.963633 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.963636 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.963685 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.963805 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.964885 (+ 1080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.964889 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.964892 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.964907 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.964942 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.964953 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.965092 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.965180 (+ 88us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.965241 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 11:37:39.965606 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.963232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:60489 (local address 127.3.240.193:45921)
0810 11:37:39.963423 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:39.963430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:39.963680 (+ 250us) server_negotiation.cc:407] Connection header received
0810 11:37:39.963858 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:39.963870 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:39.963924 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:39.964030 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:39.964860 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:39.964866 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:39.964871 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:39.964888 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:39.964932 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:39.964944 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:39.965076 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:39.965152 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 11:37:39.965215 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 11:37:39.965659 11342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.962787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36649 (local address 127.3.240.195:37669)
0810 11:37:39.963110 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.963134 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.963249 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.963842 (+ 593us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.963849 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.963873 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.964562 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.964568 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.964587 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.964650 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.965067 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.965072 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.965078 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.965192 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.965240 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":182}
I20260810 11:37:39.966084 11376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:39.962861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.3.240.195:60489)
0810 11:37:39.963587 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:39.963611 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:39.963726 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:39.964061 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:39.964067 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:39.964090 (+ 23us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:39.964666 (+ 576us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:37:39.964669 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:39.964683 (+ 14us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:37:39.964722 (+ 39us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:39.965042 (+ 320us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:39.965047 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:39.965053 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 11:37:39.965180 (+ 127us) client_negotiation.cc:240] Negotiation successful
0810 11:37:39.965230 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":551,"threads_started":1}
I20260810 11:37:39.966816 11087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbadd95a143c46fba621e108c6782a0a" candidate_uuid: "d910f08a358f40779af8f7e69f8ccd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4796049399c54856a61309fcd74524f3" is_pre_election: true
I20260810 11:37:39.967195 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbadd95a143c46fba621e108c6782a0a" candidate_uuid: "d910f08a358f40779af8f7e69f8ccd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d523ef6858854bb797839337ac31dd44" is_pre_election: true
I20260810 11:37:39.967401 11087 raft_consensus.cc:2454] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d910f08a358f40779af8f7e69f8ccd07 in term 0.
I20260810 11:37:39.967700 10904 raft_consensus.cc:2454] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d910f08a358f40779af8f7e69f8ccd07 in term 0.
I20260810 11:37:39.968140 11181 leader_election.cc:302] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4796049399c54856a61309fcd74524f3, d910f08a358f40779af8f7e69f8ccd07; no voters:
I20260810 11:37:39.968669 11375 raft_consensus.cc:2781] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:39.968827 11375 raft_consensus.cc:482] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:39.968914 11375 raft_consensus.cc:3037] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:39.971364 11375 raft_consensus.cc:504] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.972046 11375 leader_election.cc:288] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [CANDIDATE]: Term 1 election: Requested vote from peers 4796049399c54856a61309fcd74524f3 (127.3.240.194:36649), d523ef6858854bb797839337ac31dd44 (127.3.240.193:45921)
I20260810 11:37:39.972504 11087 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbadd95a143c46fba621e108c6782a0a" candidate_uuid: "d910f08a358f40779af8f7e69f8ccd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4796049399c54856a61309fcd74524f3"
I20260810 11:37:39.972579 10904 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bbadd95a143c46fba621e108c6782a0a" candidate_uuid: "d910f08a358f40779af8f7e69f8ccd07" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d523ef6858854bb797839337ac31dd44"
I20260810 11:37:39.972714 11087 raft_consensus.cc:3037] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:39.972848 10904 raft_consensus.cc:3037] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:39.975282 11087 raft_consensus.cc:2454] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d910f08a358f40779af8f7e69f8ccd07 in term 1.
I20260810 11:37:39.975720 11181 leader_election.cc:302] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4796049399c54856a61309fcd74524f3, d910f08a358f40779af8f7e69f8ccd07; no voters:
I20260810 11:37:39.975759 10904 raft_consensus.cc:2454] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d910f08a358f40779af8f7e69f8ccd07 in term 1.
I20260810 11:37:39.976020 11375 raft_consensus.cc:2781] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:39.976591 11375 raft_consensus.cc:686] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 1 LEADER]: Becoming Leader. State: Replica: d910f08a358f40779af8f7e69f8ccd07, State: Running, Role: LEADER
I20260810 11:37:39.977228 11375 consensus_queue.cc:237] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } }
I20260810 11:37:39.983559 10712 catalog_manager.cc:5495] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 reported cstate change: term changed from 0 to 1, leader changed from <none> to d910f08a358f40779af8f7e69f8ccd07 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "d910f08a358f40779af8f7e69f8ccd07" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d910f08a358f40779af8f7e69f8ccd07" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41383 } health_report { overall_health: HEALTHY } } }
I20260810 11:37:40.022755 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.020296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45372 (local address 127.3.240.195:41383)
0810 11:37:40.020519 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:40.020526 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:40.020549 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:40.020785 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:40.020789 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:40.020885 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:40.021038 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:40.022057 (+ 1019us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:40.022066 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:40.022077 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:40.022094 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:40.022128 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:40.022142 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:40.022293 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:40.022406 (+ 113us) server_negotiation.cc:299] Negotiation successful
0810 11:37:40.022472 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:37:40.033977 11087 raft_consensus.cc:1261] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 1 FOLLOWER]: Refusing update from remote peer d910f08a358f40779af8f7e69f8ccd07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:40.033978 10904 raft_consensus.cc:1261] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 1 FOLLOWER]: Refusing update from remote peer d910f08a358f40779af8f7e69f8ccd07: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:37:40.034982 11377 consensus_queue.cc:1035] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4796049399c54856a61309fcd74524f3" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:40.035391 11375 consensus_queue.cc:1035] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d523ef6858854bb797839337ac31dd44" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 45921 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 11:37:40.045318 11341 tablet.cc:2369] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:40.049582 11384 mvcc.cc:205] Tried to move back new op lower bound from 7316938178686271488 to 7316938178471088128. Current Snapshot: MvccSnapshot[applied={T|T < 7316938178686271488}]
I20260810 11:37:40.065680 10712 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:40.066025 10712 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:56706:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:37:40.069948 10712 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f06959bb5e1a428289b0ea0293f978da: Sending DeleteTablet for 3 replicas of tablet bbadd95a143c46fba621e108c6782a0a
I20260810 11:37:40.071259 10880 tablet_service.cc:1514] Processing DeleteTablet for tablet bbadd95a143c46fba621e108c6782a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:40 UTC) from {username='slave'} at 127.0.0.1:37266
I20260810 11:37:40.071511 11247 tablet_service.cc:1514] Processing DeleteTablet for tablet bbadd95a143c46fba621e108c6782a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:40 UTC) from {username='slave'} at 127.0.0.1:45362
I20260810 11:37:40.071506 11058 tablet_service.cc:1514] Processing DeleteTablet for tablet bbadd95a143c46fba621e108c6782a0a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:37:40 UTC) from {username='slave'} at 127.0.0.1:33532
I20260810 11:37:40.074460 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10785
I20260810 11:37:40.084300 11359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.076335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37280 (local address 127.3.240.193:45921)
0810 11:37:40.076528 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:40.076535 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:40.080349 (+ 3814us) server_negotiation.cc:407] Connection header received
0810 11:37:40.081307 (+ 958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:40.081314 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:40.081432 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:40.081514 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:40.083155 (+ 1641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:40.083161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:40.083166 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:40.083189 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:40.083229 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:40.083247 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:40.083362 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:40.083873 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 11:37:40.083947 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:37:40.084296 11398 tablet_replica.cc:330] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44: stopping tablet replica
I20260810 11:37:40.085372 10958 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:40.086719 11400 tablet_replica.cc:330] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: stopping tablet replica
I20260810 11:37:40.087363 11400 raft_consensus.cc:2229] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:37:40.088621 11400 raft_consensus.cc:2258] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:40.089967 11399 tablet_replica.cc:330] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: stopping tablet replica
I20260810 11:37:40.090322 11399 raft_consensus.cc:2229] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:40.090772 11399 raft_consensus.cc:2258] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:40.092700 11400 ts_tablet_manager.cc:1905] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:40.092893 11399 ts_tablet_manager.cc:1905] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:40.101051 11399 ts_tablet_manager.cc:1918] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:40.101341 11399 log.cc:1196] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/bbadd95a143c46fba621e108c6782a0a
I20260810 11:37:40.101964 11399 ts_tablet_manager.cc:1939] T bbadd95a143c46fba621e108c6782a0a P 4796049399c54856a61309fcd74524f3: Deleting consensus metadata
I20260810 11:37:40.102993 11400 ts_tablet_manager.cc:1918] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:37:40.103272 11400 log.cc:1196] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/bbadd95a143c46fba621e108c6782a0a
I20260810 11:37:40.104787 10694 catalog_manager.cc:4841] TS 4796049399c54856a61309fcd74524f3 (127.3.240.194:36649): tablet bbadd95a143c46fba621e108c6782a0a (table test-table [id=6e2313a722424357b4713fb0889e33a8]) successfully deleted
I20260810 11:37:40.105072 11400 ts_tablet_manager.cc:1939] T bbadd95a143c46fba621e108c6782a0a P d910f08a358f40779af8f7e69f8ccd07: Deleting consensus metadata
I20260810 11:37:40.106549 10694 catalog_manager.cc:4841] TS d910f08a358f40779af8f7e69f8ccd07 (127.3.240.195:41383): tablet bbadd95a143c46fba621e108c6782a0a (table test-table [id=6e2313a722424357b4713fb0889e33a8]) successfully deleted
I20260810 11:37:40.085969 11398 raft_consensus.cc:2229] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:40.221599 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10785
I20260810 11:37:40.222054 11398 raft_consensus.cc:2258] T bbadd95a143c46fba621e108c6782a0a P d523ef6858854bb797839337ac31dd44 [term 1 FOLLOWER]: Raft consensus is shut down!
W20260810 11:37:40.255028 10691 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:45921 (error 108)
I20260810 11:37:40.255578 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10972
I20260810 11:37:40.259608 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.259107 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37294)
0810 11:37:40.259319 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.259458 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 11:37:40.259722 11358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.256602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33534 (local address 127.3.240.194:36649)
0810 11:37:40.257237 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:40.257244 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:40.257266 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:37:40.257336 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:40.257342 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:40.257461 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:40.257581 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:40.258633 (+ 1052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:40.258639 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:40.258644 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:40.258662 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:40.258704 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:40.258716 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:40.259167 (+ 451us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:40.259352 (+ 185us) server_negotiation.cc:299] Negotiation successful
0810 11:37:40.259414 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":466}
W20260810 11:37:40.259913 10691 catalog_manager.cc:4584] TS d523ef6858854bb797839337ac31dd44 (127.3.240.193:45921): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet bbadd95a143c46fba621e108c6782a0a: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
I20260810 11:37:40.260515 11146 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:40.314043 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.313424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37296)
0810 11:37:40.313690 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.313817 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 11:37:40.315231 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.314813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37308)
0810 11:37:40.314996 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.315102 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 11:37:40.354406 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10972
I20260810 11:37:40.377876 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.377063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37312)
0810 11:37:40.377373 (+ 310us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.377491 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260810 11:37:40.380336 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.379810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37328)
0810 11:37:40.380064 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.380158 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128}
I20260810 11:37:40.394269 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 11157
I20260810 11:37:40.397927 11362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.395234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45382 (local address 127.3.240.195:41383)
0810 11:37:40.395579 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:40.395586 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:40.395610 (+ 24us) server_negotiation.cc:407] Connection header received
0810 11:37:40.395711 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:40.395722 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:40.395787 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:40.396158 (+ 371us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:40.397124 (+ 966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:40.397130 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:40.397135 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:40.397152 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:40.397191 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:40.397208 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:40.397355 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:40.397512 (+ 157us) server_negotiation.cc:299] Negotiation successful
0810 11:37:40.397585 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":204}
I20260810 11:37:40.399080 11333 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:40.461632 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.460668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37344)
0810 11:37:40.461105 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.461389 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20260810 11:37:40.463078 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.462511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37356)
0810 11:37:40.462778 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.462877 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
I20260810 11:37:40.526445 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 11157
I20260810 11:37:40.556576 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 10671
I20260810 11:37:40.558035 10750 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:40.675887 11363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:40.675323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:45921 (local address 127.0.0.1:37368)
0810 11:37:40.675583 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:40.675690 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:45921: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260810 11:37:40.675853 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 10671
2026-08-10T11:37:40Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3029 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260810 11:37:40.730185 4035 mini_oidc.cc:152] Starting JWKS server
I20260810 11:37:40.733992 4035 webserver.cc:460] Webserver started at https://127.0.0.1:44893/ using document root <none> and password file <none>
I20260810 11:37:40.734468 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 11:37:40.735332 4035 webserver.cc:460] Webserver started at http://127.0.0.1:44265/ using document root <none> and password file <none>
2026-08-10T11:37:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:40Z Disabled control of system clock
I20260810 11:37:40.775493 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37399
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:40549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37399
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44893/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/testchainca.cert with env {}
W20260810 11:37:41.141028 11418 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:41.141388 11418 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:41.141563 11418 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:41.149507 11418 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:41.149751 11418 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:41.149863 11418 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:41.149984 11418 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:41.150084 11418 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 11:37:41.150197 11418 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 11:37:41.150288 11418 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:44893/jwks.json
I20260810 11:37:41.162176 11418 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37399
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37399
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:44893/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:41.163658 11418 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:41.165047 11418 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:41.178409 11428 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:41.178409 11426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:41.178355 11424 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:41.180142 11418 server_base.cc:1034] running on GCE node
I20260810 11:37:41.181061 11418 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:41.182605 11418 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:41.183877 11418 hybrid_clock.cc:648] HybridClock initialized: now 1786361861183842 us; error 58 us; skew 500 ppm
I20260810 11:37:41.184286 11418 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:41.191725 11418 webserver.cc:460] Webserver started at http://127.3.240.254:46659/ using document root <none> and password file <none>
I20260810 11:37:41.192873 11418 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:41.193053 11418 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:41.193457 11418 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:41.196995 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "fa5ef11d3685440987d856a1c9450a7c"
format_stamp: "Formatted at 2026-08-10 11:37:41 on dist-test-slave-drl0"
server_key: "f6373c6aae3f5767f2015bd2ab88e44b"
server_key_iv: "634bbf0c47b67b289d1f6c18342f7768"
server_key_version: "encryptionkey@0"
I20260810 11:37:41.197896 11418 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "fa5ef11d3685440987d856a1c9450a7c"
format_stamp: "Formatted at 2026-08-10 11:37:41 on dist-test-slave-drl0"
server_key: "f6373c6aae3f5767f2015bd2ab88e44b"
server_key_iv: "634bbf0c47b67b289d1f6c18342f7768"
server_key_version: "encryptionkey@0"
I20260810 11:37:41.204075 11418 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 11:37:41.207803 11438 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:41.209287 11418 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:41.209514 11418 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "fa5ef11d3685440987d856a1c9450a7c"
format_stamp: "Formatted at 2026-08-10 11:37:41 on dist-test-slave-drl0"
server_key: "f6373c6aae3f5767f2015bd2ab88e44b"
server_key_iv: "634bbf0c47b67b289d1f6c18342f7768"
server_key_version: "encryptionkey@0"
I20260810 11:37:41.209748 11418 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:41.231994 11418 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:41.233062 11418 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:41.233451 11418 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:41.258679 11418 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37399
I20260810 11:37:41.258708 11499 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37399 every 8 connection(s)
I20260810 11:37:41.260310 11418 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:41.261503 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 11418
I20260810 11:37:41.261749 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:41.262542 4035 external_mini_cluster.cc:1428] Setting key dc1d164084157d4dd82b71f881a2ce61
I20260810 11:37:41.266501 11501 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:41.279325 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c: Bootstrap starting.
I20260810 11:37:41.279753 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:41.265356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55116 (local address 127.3.240.254:37399)
0810 11:37:41.266764 (+ 1408us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:41.266778 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:41.266828 (+ 50us) server_negotiation.cc:407] Connection header received
0810 11:37:41.268962 (+ 2134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:41.268990 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:41.269556 (+ 566us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:41.270064 (+ 508us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:41.270777 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.272482 (+ 1705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.274444 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.275392 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.275754 (+ 362us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:41.277211 (+ 1457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:41.277220 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:41.277236 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:41.277788 (+ 552us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:41.277874 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:41.277902 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:41.278033 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:41.278304 (+ 271us) server_negotiation.cc:299] Negotiation successful
0810 11:37:41.278642 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":787,"thread_start_us":427,"threads_started":1}
I20260810 11:37:41.283394 11501 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:41.284384 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:37:41.284686 11501 log.cc:824] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:41.287859 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c: No bootstrap required, opened a new log
I20260810 11:37:41.296618 11501 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } }
I20260810 11:37:41.297010 11501 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:41.297178 11501 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa5ef11d3685440987d856a1c9450a7c, State: Initialized, Role: FOLLOWER
I20260810 11:37:41.297833 11501 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } }
I20260810 11:37:41.298053 11501 raft_consensus.cc:388] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:41.298154 11501 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:41.298282 11501 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:41.300436 11501 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } }
I20260810 11:37:41.300983 11501 leader_election.cc:302] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fa5ef11d3685440987d856a1c9450a7c; no voters:
I20260810 11:37:41.301648 11501 leader_election.cc:288] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:41.301841 11506 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:41.303063 11506 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [term 1 LEADER]: Becoming Leader. State: Replica: fa5ef11d3685440987d856a1c9450a7c, State: Running, Role: LEADER
I20260810 11:37:41.303586 11506 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } }
I20260810 11:37:41.304044 11501 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:41.306862 11507 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fa5ef11d3685440987d856a1c9450a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } } }
I20260810 11:37:41.307075 11508 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa5ef11d3685440987d856a1c9450a7c. Latest consensus state: current_term: 1 leader_uuid: "fa5ef11d3685440987d856a1c9450a7c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fa5ef11d3685440987d856a1c9450a7c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37399 } } }
I20260810 11:37:41.307514 11507 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:41.307710 11508 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:41.308876 11517 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:41.316233 11517 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:41.325739 11517 catalog_manager.cc:1347] Generated new cluster ID: c3c278c825ba41c38215fe6565047577
I20260810 11:37:41.325862 11517 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:41.346570 11517 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:41.348482 11517 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:41.362649 11517 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa5ef11d3685440987d856a1c9450a7c: Generated new TSK 0
I20260810 11:37:41.364059 11517 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 11:37:41.407265 11550 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260810 11:37:41.407596 11550 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 11:37:41.412075 11504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:41.384907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55126 (local address 127.3.240.254:37399)
0810 11:37:41.385127 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:41.385134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:41.385661 (+ 527us) server_negotiation.cc:407] Connection header received
0810 11:37:41.386000 (+ 339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:41.386006 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:41.386106 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:41.386269 (+ 163us) server_negotiation.cc:226] Negotiated authn=JWT
0810 11:37:41.386887 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.388074 (+ 1187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.389532 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.390223 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.390379 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:41.390971 (+ 592us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 11:37:41.411323 (+ 20352us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 11:37:41.411718 (+ 395us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":69}
W20260810 11:37:41.412210 11504 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55126: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260810 11:37:41.413846 11548 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:41.384808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37399 (local address 127.0.0.1:55126)
0810 11:37:41.385554 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:41.385572 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:41.385721 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:41.386336 (+ 615us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:41.386340 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:41.386352 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0810 11:37:41.386702 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:41.386712 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.388253 (+ 1541us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.388257 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:41.389391 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:41.389402 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:41.390453 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:41.390458 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:41.390589 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:41.390649 (+ 60us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 11:37:41.411670 (+ 21021us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0810 11:37:41.411973 (+ 303us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:37399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":391,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.3.240.254:37399: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:44893/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260810 11:37:41.415333 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 11418
I20260810 11:37:41.416603 11494 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:41.516177 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 11418
2026-08-10T11:37:41Z chronyd exiting
I20260810 11:37:41.578168 4035 test_util.cc:175] -----------------------------------------------
I20260810 11:37:41.578336 4035 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786361809851056-4035-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (853 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260810 11:37:41.582181 4035 mini_oidc.cc:152] Starting JWKS server
I20260810 11:37:41.584344 4035 webserver.cc:460] Webserver started at https://127.0.0.1:33277/ using document root <none> and password file <none>
I20260810 11:37:41.584592 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 11:37:41.585361 4035 webserver.cc:460] Webserver started at http://127.0.0.1:37605/ using document root <none> and password file <none>
2026-08-10T11:37:41Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:41Z Disabled control of system clock
I20260810 11:37:41.625451 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37329
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:35941
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:37329
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33277/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/testchainca.cert with env {}
W20260810 11:37:41.996984 11564 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:41.997349 11564 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:41.997531 11564 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:42.005223 11564 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:42.005415 11564 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:42.005529 11564 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:42.005632 11564 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:42.005739 11564 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 11:37:42.005846 11564 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 11:37:42.005934 11564 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:33277/jwks.json
I20260810 11:37:42.016438 11564 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35941
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:37329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:37329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:33277/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:42.017850 11564 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:42.019380 11564 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:42.033396 11569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:42.033363 11570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:42.033329 11573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:42.034250 11564 server_base.cc:1034] running on GCE node
I20260810 11:37:42.035137 11564 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:42.036737 11564 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:42.038053 11564 hybrid_clock.cc:648] HybridClock initialized: now 1786361862037967 us; error 51 us; skew 500 ppm
I20260810 11:37:42.038630 11564 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:42.041808 11564 webserver.cc:460] Webserver started at http://127.3.240.254:38755/ using document root <none> and password file <none>
I20260810 11:37:42.042882 11564 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:42.043061 11564 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:42.043439 11564 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:42.047478 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "ea28713e5a724db396a74b61c26e3d4d"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "46c27aec5976cd29c1f77c1631eb4fd2"
server_key_iv: "04f8e5e6fe876651cc301398769925ef"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.048431 11564 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "ea28713e5a724db396a74b61c26e3d4d"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "46c27aec5976cd29c1f77c1631eb4fd2"
server_key_iv: "04f8e5e6fe876651cc301398769925ef"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.054347 11564 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:37:42.058054 11583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:42.060251 11564 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:42.060556 11564 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "ea28713e5a724db396a74b61c26e3d4d"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "46c27aec5976cd29c1f77c1631eb4fd2"
server_key_iv: "04f8e5e6fe876651cc301398769925ef"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.060827 11564 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:42.088581 11564 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:42.089954 11564 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:42.090369 11564 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:42.112371 11564 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:37329
I20260810 11:37:42.112409 11650 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:37329 every 8 connection(s)
I20260810 11:37:42.114434 11564 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:42.118813 11651 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:42.125130 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 11564
I20260810 11:37:42.125401 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:42.126361 4035 external_mini_cluster.cc:1428] Setting key 6ce850c6735ce703ebdd563c1bc165f8
I20260810 11:37:42.130319 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d: Bootstrap starting.
I20260810 11:37:42.134125 11651 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:42.135444 11651 log.cc:824] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:42.138346 11651 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d: No bootstrap required, opened a new log
I20260810 11:37:42.144146 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:42.128935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48026 (local address 127.3.240.254:37329)
0810 11:37:42.130275 (+ 1340us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:42.130294 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:42.130373 (+ 79us) server_negotiation.cc:407] Connection header received
0810 11:37:42.132885 (+ 2512us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:42.132913 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:42.133375 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:42.133787 (+ 412us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:42.134362 (+ 575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.136208 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.137803 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.138754 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.139051 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:42.140700 (+ 1649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:42.140709 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:42.140730 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:42.141550 (+ 820us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:42.141664 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:42.141702 (+ 38us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:42.141896 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:42.142197 (+ 301us) server_negotiation.cc:299] Negotiation successful
0810 11:37:42.142633 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":386,"threads_started":1}
I20260810 11:37:42.147809 11651 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } }
I20260810 11:37:42.148196 11651 raft_consensus.cc:374] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:42.148325 11651 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea28713e5a724db396a74b61c26e3d4d, State: Initialized, Role: FOLLOWER
I20260810 11:37:42.149036 11651 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } }
I20260810 11:37:42.149441 11651 raft_consensus.cc:388] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:42.149564 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:37:42.149610 11651 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:42.149799 11651 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:42.153084 11651 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } }
I20260810 11:37:42.153628 11651 leader_election.cc:302] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ea28713e5a724db396a74b61c26e3d4d; no voters:
I20260810 11:37:42.154275 11651 leader_election.cc:288] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:42.154413 11657 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:42.155246 11657 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [term 1 LEADER]: Becoming Leader. State: Replica: ea28713e5a724db396a74b61c26e3d4d, State: Running, Role: LEADER
I20260810 11:37:42.155874 11657 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } }
I20260810 11:37:42.156554 11651 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:42.159042 11660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea28713e5a724db396a74b61c26e3d4d. Latest consensus state: current_term: 1 leader_uuid: "ea28713e5a724db396a74b61c26e3d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } } }
I20260810 11:37:42.159598 11660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:42.159437 11658 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ea28713e5a724db396a74b61c26e3d4d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ea28713e5a724db396a74b61c26e3d4d" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 37329 } } }
I20260810 11:37:42.159814 11658 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:42.162014 11668 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:42.169487 11668 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:42.181767 11668 catalog_manager.cc:1347] Generated new cluster ID: 716531b1b3984b14beea80ba1632a3e8
I20260810 11:37:42.181945 11668 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:42.209162 11668 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:42.210744 11668 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:42.221735 11668 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea28713e5a724db396a74b61c26e3d4d: Generated new TSK 0
I20260810 11:37:42.222642 11668 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260810 11:37:42.268538 11702 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260810 11:37:42.268682 11702 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 11:37:42.270066 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:42.240657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48030 (local address 127.3.240.254:37329)
0810 11:37:42.240965 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:42.240972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:42.241280 (+ 308us) server_negotiation.cc:407] Connection header received
0810 11:37:42.241587 (+ 307us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:42.241592 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:42.241668 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:42.241862 (+ 194us) server_negotiation.cc:226] Negotiated authn=JWT
0810 11:37:42.242399 (+ 537us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.243262 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.244881 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.245395 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.245564 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:42.246226 (+ 662us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 11:37:42.269350 (+ 23124us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 11:37:42.269787 (+ 437us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":126}
W20260810 11:37:42.270284 11654 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48030: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260810 11:37:42.270470 11700 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:42.240483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:37329 (local address 127.0.0.1:48030)
0810 11:37:42.241149 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:42.241175 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:42.241370 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:42.241876 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:42.241882 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:42.241895 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0810 11:37:42.242214 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:42.242225 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.243431 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.243440 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:42.244596 (+ 1156us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:42.244608 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.245610 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.245618 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:42.245795 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:42.245859 (+ 64us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 11:37:42.269688 (+ 23829us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0810 11:37:42.270028 (+ 340us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:37329: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:33277/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":337,"threads_started":1}
I20260810 11:37:42.273177 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 11564
I20260810 11:37:42.274820 11645 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:42.374060 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 11564
2026-08-10T11:37:42Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (856 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260810 11:37:42.438501 4035 mini_oidc.cc:152] Starting JWKS server
I20260810 11:37:42.440496 4035 webserver.cc:460] Webserver started at https://127.0.0.1:36001/ using document root <none> and password file <none>
I20260810 11:37:42.440698 4035 mini_oidc.cc:189] Starting OIDC Discovery server
I20260810 11:37:42.441296 4035 webserver.cc:460] Webserver started at http://127.0.0.1:33579/ using document root <none> and password file <none>
2026-08-10T11:37:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:42Z Disabled control of system clock
I20260810 11:37:42.481062 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40213
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:43927
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40213
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 11:37:42.848119 11710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:42.848424 11710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:42.848579 11710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:42.856799 11710 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:42.856904 11710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:42.856985 11710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:42.857069 11710 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:42.857108 11710 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 11:37:42.857219 11710 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 11:37:42.857270 11710 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:36001/jwks.json
I20260810 11:37:42.867415 11710 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43927
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40213
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:36001/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:42.868793 11710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:42.870625 11710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:42.884895 11721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:42.885007 11717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:42.885169 11719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:42.885660 11710 server_base.cc:1034] running on GCE node
I20260810 11:37:42.886515 11710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:42.888228 11710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:42.889693 11710 hybrid_clock.cc:648] HybridClock initialized: now 1786361862889633 us; error 88 us; skew 500 ppm
I20260810 11:37:42.890093 11710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:42.892736 11710 webserver.cc:460] Webserver started at http://127.3.240.254:37913/ using document root <none> and password file <none>
I20260810 11:37:42.893487 11710 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:42.893630 11710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:42.893988 11710 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:42.896998 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "3c2cb8edcf7944f9a5654d312279a0a7"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "fc5e8ebb99a78d71d801b074a1f0fa81"
server_key_iv: "ef87c08a6e4c2700ed4070e53e391e0e"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.897840 11710 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "3c2cb8edcf7944f9a5654d312279a0a7"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "fc5e8ebb99a78d71d801b074a1f0fa81"
server_key_iv: "ef87c08a6e4c2700ed4070e53e391e0e"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.903578 11710 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 11:37:42.908085 11728 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:42.909832 11710 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:42.910127 11710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "3c2cb8edcf7944f9a5654d312279a0a7"
format_stamp: "Formatted at 2026-08-10 11:37:42 on dist-test-slave-drl0"
server_key: "fc5e8ebb99a78d71d801b074a1f0fa81"
server_key_iv: "ef87c08a6e4c2700ed4070e53e391e0e"
server_key_version: "encryptionkey@0"
I20260810 11:37:42.910363 11710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:42.923290 11710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:42.924089 11710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:42.924414 11710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:42.951615 11710 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:40213
I20260810 11:37:42.951651 11796 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:40213 every 8 connection(s)
I20260810 11:37:42.952996 11710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:42.957167 11797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:42.961645 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 11710
I20260810 11:37:42.961884 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:42.962731 4035 external_mini_cluster.cc:1428] Setting key d674a491b38da75bf22b9a5e8bdad0ab
I20260810 11:37:42.969544 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7: Bootstrap starting.
I20260810 11:37:42.973485 11797 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:42.974749 11797 log.cc:824] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:42.977017 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:42.964842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48934 (local address 127.3.240.254:40213)
0810 11:37:42.965831 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:42.965841 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:42.965889 (+ 48us) server_negotiation.cc:407] Connection header received
0810 11:37:42.967897 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:42.967922 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:42.968331 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:42.968703 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:42.969259 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.970590 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.971988 (+ 1398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:42.972642 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:42.972842 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:42.974208 (+ 1366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:42.974216 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:42.974228 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:42.974814 (+ 586us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:42.974905 (+ 91us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:42.975000 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:42.975130 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:42.975393 (+ 263us) server_negotiation.cc:299] Negotiation successful
0810 11:37:42.975696 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":366,"threads_started":1}
I20260810 11:37:42.978788 11797 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7: No bootstrap required, opened a new log
I20260810 11:37:42.982254 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:37:42.987123 11797 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } }
I20260810 11:37:42.987504 11797 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:42.987612 11797 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3c2cb8edcf7944f9a5654d312279a0a7, State: Initialized, Role: FOLLOWER
I20260810 11:37:42.988406 11797 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } }
I20260810 11:37:42.989084 11797 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:42.989295 11797 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:42.989471 11797 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:42.992755 11797 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } }
I20260810 11:37:42.993492 11797 leader_election.cc:302] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3c2cb8edcf7944f9a5654d312279a0a7; no voters:
I20260810 11:37:42.994170 11797 leader_election.cc:288] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:42.994282 11804 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:42.995268 11804 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [term 1 LEADER]: Becoming Leader. State: Replica: 3c2cb8edcf7944f9a5654d312279a0a7, State: Running, Role: LEADER
I20260810 11:37:42.995888 11804 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } }
I20260810 11:37:42.997411 11797 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:43.000268 11807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3c2cb8edcf7944f9a5654d312279a0a7. Latest consensus state: current_term: 1 leader_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } } }
I20260810 11:37:43.001057 11807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:42.999729 11806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3c2cb8edcf7944f9a5654d312279a0a7" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40213 } } }
I20260810 11:37:43.002090 11806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:43.003798 11814 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:43.010947 11814 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:43.022557 11814 catalog_manager.cc:1347] Generated new cluster ID: a43a9e5d1c424f37ac633c71f11093cb
I20260810 11:37:43.022691 11814 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:43.039067 11814 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:43.040284 11814 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:43.048879 11814 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3c2cb8edcf7944f9a5654d312279a0a7: Generated new TSK 0
I20260810 11:37:43.049779 11814 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:37:43.085953 11800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:43.074666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48948 (local address 127.3.240.254:40213)
0810 11:37:43.074866 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:43.074874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:43.075563 (+ 689us) server_negotiation.cc:407] Connection header received
0810 11:37:43.076006 (+ 443us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:43.076011 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:43.076112 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:43.076335 (+ 223us) server_negotiation.cc:226] Negotiated authn=JWT
0810 11:37:43.077146 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.078590 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.080286 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.081195 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.081392 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:43.082074 (+ 682us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 11:37:43.085202 (+ 3128us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 11:37:43.085590 (+ 388us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:48948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":57}
W20260810 11:37:43.086094 11800 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:48948: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260810 11:37:43.087740 11838 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:43.074617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:40213 (local address 127.0.0.1:48948)
0810 11:37:43.075254 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:43.075270 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:43.075719 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:43.076354 (+ 635us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:43.076360 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:43.076375 (+ 15us) client_negotiation.cc:189] Negotiated authn=JWT
0810 11:37:43.076840 (+ 465us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:43.076860 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.078915 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.078920 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:43.080127 (+ 1207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:43.080139 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.081482 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.081488 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:43.081621 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:43.081658 (+ 37us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 11:37:43.085466 (+ 3808us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0810 11:37:43.087291 (+ 1825us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:40213: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:36001/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":1503,"thread_start_us":375,"threads_started":1}
I20260810 11:37:43.088728 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 11710
I20260810 11:37:43.089982 11791 generic_service.cc:196] Checking for leaks (request via RPC)
E20260810 11:37:43.098451 11841 webserver.cc:533] Webserver: sslize failed:
E20260810 11:37:43.098591 11841 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260810 11:37:43.185552 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 11710
2026-08-10T11:37:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (808 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-10T11:37:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:43Z Disabled control of system clock
I20260810 11:37:43.288678 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35795
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:34627
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35795
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/testchainca.cert with env {}
W20260810 11:37:43.659750 11850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:43.660043 11850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:43.660203 11850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:43.667570 11850 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:43.667667 11850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:43.667761 11850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:43.667914 11850 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:43.668087 11850 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260810 11:37:43.668208 11850 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260810 11:37:43.668268 11850 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260810 11:37:43.678362 11850 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34627
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35795
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:43.679723 11850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:43.681921 11850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:43.695297 11862 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:43.695585 11860 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:43.695865 11859 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:43.697170 11850 server_base.cc:1034] running on GCE node
I20260810 11:37:43.698249 11850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:43.700269 11850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:43.701570 11850 hybrid_clock.cc:648] HybridClock initialized: now 1786361863701503 us; error 49 us; skew 500 ppm
I20260810 11:37:43.702256 11850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:37:43.705682 11850 webserver.cc:460] Webserver started at http://127.3.240.254:42971/ using document root <none> and password file <none>
I20260810 11:37:43.706463 11850 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:43.706604 11850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:43.706992 11850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:43.710000 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "e67bcd0b751d4c048fed905afc78a6be"
format_stamp: "Formatted at 2026-08-10 11:37:43 on dist-test-slave-drl0"
server_key: "f3d524f87f607cc39773c00779623ce7"
server_key_iv: "9b8f8784ff1136c5dacd01410a3e4540"
server_key_version: "encryptionkey@0"
I20260810 11:37:43.710813 11850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "e67bcd0b751d4c048fed905afc78a6be"
format_stamp: "Formatted at 2026-08-10 11:37:43 on dist-test-slave-drl0"
server_key: "f3d524f87f607cc39773c00779623ce7"
server_key_iv: "9b8f8784ff1136c5dacd01410a3e4540"
server_key_version: "encryptionkey@0"
I20260810 11:37:43.716243 11850 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 11:37:43.720573 11870 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:43.722412 11850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:37:43.722723 11850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "e67bcd0b751d4c048fed905afc78a6be"
format_stamp: "Formatted at 2026-08-10 11:37:43 on dist-test-slave-drl0"
server_key: "f3d524f87f607cc39773c00779623ce7"
server_key_iv: "9b8f8784ff1136c5dacd01410a3e4540"
server_key_version: "encryptionkey@0"
I20260810 11:37:43.723218 11850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:43.743799 11850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:43.745019 11850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:43.745491 11850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:43.768132 11850 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35795
I20260810 11:37:43.768164 11950 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35795 every 8 connection(s)
I20260810 11:37:43.769589 11850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:43.773525 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 11850
I20260810 11:37:43.773729 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:43.774453 11952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:43.774606 4035 external_mini_cluster.cc:1428] Setting key d9ff0ed2554a56e9bd59ea2d534816cd
I20260810 11:37:43.789682 11952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be: Bootstrap starting.
I20260810 11:37:43.790833 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:43.776679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46916 (local address 127.3.240.254:35795)
0810 11:37:43.777873 (+ 1194us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:43.777886 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:43.777932 (+ 46us) server_negotiation.cc:407] Connection header received
0810 11:37:43.780147 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:43.780186 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:43.780906 (+ 720us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:43.781393 (+ 487us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:43.782039 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.783774 (+ 1735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.785690 (+ 1916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.786426 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.786579 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:43.787957 (+ 1378us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:43.787967 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:43.787985 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:37:43.788722 (+ 737us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:43.788835 (+ 113us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:37:43.788864 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:43.789002 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:43.789311 (+ 309us) server_negotiation.cc:299] Negotiation successful
0810 11:37:43.789865 (+ 554us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738,"thread_start_us":500,"threads_started":1}
I20260810 11:37:43.793736 11952 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:43.794915 11952 log.cc:824] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:43.795017 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:37:43.798699 11952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be: No bootstrap required, opened a new log
I20260810 11:37:43.806406 11952 raft_consensus.cc:348] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } }
I20260810 11:37:43.806849 11952 raft_consensus.cc:374] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:43.807056 11952 raft_consensus.cc:729] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e67bcd0b751d4c048fed905afc78a6be, State: Initialized, Role: FOLLOWER
I20260810 11:37:43.807669 11952 consensus_queue.cc:260] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } }
I20260810 11:37:43.807962 11952 raft_consensus.cc:388] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:43.808108 11952 raft_consensus.cc:482] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:43.808293 11952 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:43.810415 11952 raft_consensus.cc:504] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } }
I20260810 11:37:43.811090 11952 leader_election.cc:302] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e67bcd0b751d4c048fed905afc78a6be; no voters:
I20260810 11:37:43.811720 11952 leader_election.cc:288] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:43.811940 11963 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:43.812790 11963 raft_consensus.cc:686] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [term 1 LEADER]: Becoming Leader. State: Replica: e67bcd0b751d4c048fed905afc78a6be, State: Running, Role: LEADER
I20260810 11:37:43.813350 11963 consensus_queue.cc:237] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } }
I20260810 11:37:43.814087 11952 sys_catalog.cc:564] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:43.816764 11964 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e67bcd0b751d4c048fed905afc78a6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } } }
I20260810 11:37:43.816744 11965 sys_catalog.cc:455] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [sys.catalog]: SysCatalogTable state changed. Reason: New leader e67bcd0b751d4c048fed905afc78a6be. Latest consensus state: current_term: 1 leader_uuid: "e67bcd0b751d4c048fed905afc78a6be" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e67bcd0b751d4c048fed905afc78a6be" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35795 } } }
I20260810 11:37:43.817178 11964 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:43.817243 11965 sys_catalog.cc:458] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:43.820916 11971 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:43.826297 11971 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:43.836522 11971 catalog_manager.cc:1347] Generated new cluster ID: e301113ad0e24dc99afad075e92ea578
I20260810 11:37:43.836649 11971 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:43.878203 11971 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:43.879721 11971 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:43.891819 11971 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e67bcd0b751d4c048fed905afc78a6be: Generated new TSK 0
I20260810 11:37:43.892866 11971 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:37:43.989194 11961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:43.911808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46922 (local address 127.3.240.254:35795)
0810 11:37:43.912104 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:43.912114 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:43.912589 (+ 475us) server_negotiation.cc:407] Connection header received
0810 11:37:43.912913 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:43.912918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:43.913014 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:43.913209 (+ 195us) server_negotiation.cc:226] Negotiated authn=JWT
0810 11:37:43.913888 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.914743 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.916829 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.917301 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.917415 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:43.918014 (+ 599us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0810 11:37:43.988425 (+ 70411us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 11:37:43.988874 (+ 449us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":90}
W20260810 11:37:43.989351 11961 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46922: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260810 11:37:43.989703 11990 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:43.911603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35795 (local address 127.0.0.1:46922)
0810 11:37:43.912426 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:43.912448 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:43.912626 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:43.913244 (+ 618us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:43.913251 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:43.913268 (+ 17us) client_negotiation.cc:189] Negotiated authn=JWT
0810 11:37:43.913655 (+ 387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:43.913667 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.914912 (+ 1245us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.914918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:43.916665 (+ 1747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:43.916679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:43.917488 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:43.917494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:43.917679 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:43.917725 (+ 46us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0810 11:37:43.988785 (+ 71060us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0810 11:37:43.989109 (+ 324us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.3.240.254:35795: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":662,"thread_start_us":516,"threads_started":1}
I20260810 11:37:43.991396 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 11850
I20260810 11:37:43.992630 11946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:44.102808 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 11850
2026-08-10T11:37:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (898 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (466 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1746 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:46 dist-test-slave-drl0 krb5kdc[12018](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:46 dist-test-slave-drl0 krb5kdc[12018](info): set up 2 sockets
Aug 10 11:37:46 dist-test-slave-drl0 krb5kdc[12018](info): commencing operation
krb5kdc: starting...
W20260810 11:37:47.657894 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.003s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:47 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:37:47.946178 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34593
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:34593
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:48.319790 12035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:48.320241 12035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:48.320421 12035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:48.329418 12035 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:48.329597 12035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:48.329712 12035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:48.329828 12035 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:48.329909 12035 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:48.340457 12035 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:34593
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:34593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:48.341869 12035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:48.343298 12035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:48.356559 12047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:48.356623 12050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:48.356606 12048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:48.358325 12035 server_base.cc:1034] running on GCE node
I20260810 11:37:48.359225 12035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:48.360971 12035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:48.362267 12035 hybrid_clock.cc:648] HybridClock initialized: now 1786361868362199 us; error 48 us; skew 500 ppm
I20260810 11:37:48.362653 12035 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:48 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361868, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:48.366068 12035 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:37:48.367848 12035 webserver.cc:460] Webserver started at http://127.3.240.254:42153/ using document root <none> and password file <none>
I20260810 11:37:48.369378 12035 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:48.369688 12035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:48.370082 12035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:48.373440 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "fb38d9f177474e93b91b16b445a0ae1b"
format_stamp: "Formatted at 2026-08-10 11:37:48 on dist-test-slave-drl0"
server_key: "b41c0ae41cfa8da4a9d1a368ad36cea0"
server_key_iv: "30a1a596e836cabb7fea7ec2ec292ecb"
server_key_version: "encryptionkey@0"
I20260810 11:37:48.374387 12035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "fb38d9f177474e93b91b16b445a0ae1b"
format_stamp: "Formatted at 2026-08-10 11:37:48 on dist-test-slave-drl0"
server_key: "b41c0ae41cfa8da4a9d1a368ad36cea0"
server_key_iv: "30a1a596e836cabb7fea7ec2ec292ecb"
server_key_version: "encryptionkey@0"
I20260810 11:37:48.380337 12035 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:37:48.384487 12060 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:48.386044 12035 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.002s
I20260810 11:37:48.386279 12035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "fb38d9f177474e93b91b16b445a0ae1b"
format_stamp: "Formatted at 2026-08-10 11:37:48 on dist-test-slave-drl0"
server_key: "b41c0ae41cfa8da4a9d1a368ad36cea0"
server_key_iv: "30a1a596e836cabb7fea7ec2ec292ecb"
server_key_version: "encryptionkey@0"
I20260810 11:37:48.386512 12035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:48.413760 12035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:48.418953 12035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:48.419298 12035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:48.440124 12035 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:34593
I20260810 11:37:48.440168 12131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:34593 every 8 connection(s)
I20260810 11:37:48.441531 12035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:48.445040 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12035
I20260810 11:37:48.445369 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:48.445672 12132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:48.446125 4035 external_mini_cluster.cc:1428] Setting key 9e3620ce36d0a78e83fb8942871ce48a
I20260810 11:37:48.459890 12132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b: Bootstrap starting.
Aug 10 11:37:48 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361867, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:48.463477 12132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:48.464553 12132 log.cc:824] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:48.467780 12132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b: No bootstrap required, opened a new log
I20260810 11:37:48.471418 12135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:48.448356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52954 (local address 127.3.240.254:34593)
0810 11:37:48.449212 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:48.449223 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:48.449263 (+ 40us) server_negotiation.cc:407] Connection header received
0810 11:37:48.451522 (+ 2259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:48.451543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:48.451925 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:48.452248 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:48.454035 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:48.455392 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:48.457395 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:48.458101 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:48.458258 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:48.463254 (+ 4996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:48.463263 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:48.463277 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:48.463322 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:48.466838 (+ 3516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:48.467614 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:48.467621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:48.467626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:48.467818 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:48.468300 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:48.468306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:48.468310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:48.468935 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:48.469278 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:48.469851 (+ 573us) server_negotiation.cc:299] Negotiation successful
0810 11:37:48.470287 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":403,"threads_started":1}
I20260810 11:37:48.476372 12132 raft_consensus.cc:348] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } }
I20260810 11:37:48.476747 12132 raft_consensus.cc:374] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:48.476850 12132 raft_consensus.cc:729] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb38d9f177474e93b91b16b445a0ae1b, State: Initialized, Role: FOLLOWER
I20260810 11:37:48.477476 12132 consensus_queue.cc:260] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } }
I20260810 11:37:48.477711 12132 raft_consensus.cc:388] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:48.477840 12132 raft_consensus.cc:482] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:48.478044 12132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:48.481459 12132 raft_consensus.cc:504] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } }
I20260810 11:37:48.482354 12132 leader_election.cc:302] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fb38d9f177474e93b91b16b445a0ae1b; no voters:
I20260810 11:37:48.483047 12132 leader_election.cc:288] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:48.483242 12137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:48.484148 12137 raft_consensus.cc:686] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [term 1 LEADER]: Becoming Leader. State: Replica: fb38d9f177474e93b91b16b445a0ae1b, State: Running, Role: LEADER
I20260810 11:37:48.484865 12137 consensus_queue.cc:237] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } }
I20260810 11:37:48.485566 12132 sys_catalog.cc:564] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:48.488605 12139 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader fb38d9f177474e93b91b16b445a0ae1b. Latest consensus state: current_term: 1 leader_uuid: "fb38d9f177474e93b91b16b445a0ae1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } } }
I20260810 11:37:48.489037 12139 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:48.489913 12138 sys_catalog.cc:455] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fb38d9f177474e93b91b16b445a0ae1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fb38d9f177474e93b91b16b445a0ae1b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 34593 } } }
I20260810 11:37:48.490304 12138 sys_catalog.cc:458] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:48.492940 12145 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:48.498855 12145 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:48.509676 12145 catalog_manager.cc:1347] Generated new cluster ID: 130c29a17258461f88988129eff5e623
I20260810 11:37:48.509814 12145 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:48.531204 12145 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:48.533594 12145 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:48.549944 12145 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fb38d9f177474e93b91b16b445a0ae1b: Generated new TSK 0
I20260810 11:37:48.551059 12145 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:37:48.619585 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:48.986060 12167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:48.986342 12167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:48.986505 12167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:48.994421 12167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:48.994529 12167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:48.994735 12167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:49.004698 12167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:49.005995 12167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:49.007563 12167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:49.021409 12180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:49.021461 12185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:49.021502 12179 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:49.022419 12167 server_base.cc:1034] running on GCE node
I20260810 11:37:49.023523 12167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:49.024596 12167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:49.025861 12167 hybrid_clock.cc:648] HybridClock initialized: now 1786361869025803 us; error 42 us; skew 500 ppm
I20260810 11:37:49.026208 12167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:49 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:49.029742 12167 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:37:49.031421 12167 webserver.cc:460] Webserver started at http://127.3.240.193:39665/ using document root <none> and password file <none>
I20260810 11:37:49.032140 12167 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:49.032305 12167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:49.032655 12167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:49.035856 12167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "2deee972d301483595ba1ad55769760d"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "38bef7dc9807b56367b34b1dc5056de5"
server_key_iv: "769257c7fb2fd2da393427cd9038f1ba"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.036712 12167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "2deee972d301483595ba1ad55769760d"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "38bef7dc9807b56367b34b1dc5056de5"
server_key_iv: "769257c7fb2fd2da393427cd9038f1ba"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.042366 12167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:49.046561 12192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.048291 12167 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260810 11:37:49.048497 12167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "2deee972d301483595ba1ad55769760d"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "38bef7dc9807b56367b34b1dc5056de5"
server_key_iv: "769257c7fb2fd2da393427cd9038f1ba"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.048725 12167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:49.078122 12167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:49.083695 12167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:49.084015 12167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:49.085029 12167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:49.087384 12167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:49.087504 12167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.087656 12167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:49.087728 12167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.128809 12167 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:32931
I20260810 11:37:49.128846 12359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:32931 every 8 connection(s)
I20260810 11:37:49.130231 12167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:49.130788 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12167
I20260810 11:37:49.131099 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:49.132081 4035 external_mini_cluster.cc:1428] Setting key 1294ddf6b22d9f494d996137ef2f47cf
Aug 10 11:37:49 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:49.155086 12365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:49.133769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:43847 (local address 127.3.240.254:34593)
0810 11:37:49.134293 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:49.134299 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:49.135503 (+ 1204us) server_negotiation.cc:407] Connection header received
0810 11:37:49.137347 (+ 1844us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:49.137352 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:49.137442 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:49.137558 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:49.139080 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.140189 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.141789 (+ 1600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.142198 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.142335 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:49.146475 (+ 4140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:49.146485 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:49.146493 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:49.146539 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:49.150279 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:49.151629 (+ 1350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:49.151637 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:49.151643 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:49.151799 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:49.152482 (+ 683us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:49.152490 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:49.152496 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:49.152939 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:49.153118 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:49.154031 (+ 913us) server_negotiation.cc:299] Negotiation successful
0810 11:37:49.154310 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":249,"threads_started":1}
I20260810 11:37:49.155472 12366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:49.134417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34593 (local address 127.3.240.193:43847)
0810 11:37:49.135350 (+ 933us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:49.135399 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:49.137089 (+ 1690us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:49.137782 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:49.137790 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:49.138292 (+ 502us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:49.138858 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:49.138875 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.140361 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.140367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:49.141639 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:49.141652 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.142445 (+ 793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.142454 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:49.142889 (+ 435us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:49.143695 (+ 806us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:49.143718 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:49.146235 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:49.150573 (+ 4338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:49.150585 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:49.150606 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:49.151440 (+ 834us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:49.151995 (+ 555us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:49.152000 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:49.152003 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:49.152320 (+ 317us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:49.153125 (+ 805us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:49.153133 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:49.153542 (+ 409us) client_negotiation.cc:769] Sending connection context
0810 11:37:49.153828 (+ 286us) client_negotiation.cc:240] Negotiation successful
0810 11:37:49.154249 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":458,"threads_started":1}
I20260810 11:37:49.157155 12362 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34593
I20260810 11:37:49.157496 12362 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:49.158241 12362 heartbeater.cc:511] Master 127.3.240.254:34593 requested a full tablet report, sending...
I20260810 11:37:49.160388 12083 ts_manager.cc:194] Registered new tserver with Master: 2deee972d301483595ba1ad55769760d (127.3.240.193:32931)
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:37:49.161969 12083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:43847
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:37:49.194344 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:49.551206 12371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:49.551487 12371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:49.551590 12371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:49.560199 12371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:49.560442 12371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:49.560637 12371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:49.571357 12371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:49.572656 12371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:49.573953 12371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:49.589076 12376 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:49.588999 12378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:49.589037 12382 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:49.590227 12371 server_base.cc:1034] running on GCE node
I20260810 11:37:49.591032 12371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:49.592125 12371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:49.593384 12371 hybrid_clock.cc:648] HybridClock initialized: now 1786361869593283 us; error 56 us; skew 500 ppm
I20260810 11:37:49.593730 12371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:49 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:49.597293 12371 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:37:49.599004 12371 webserver.cc:460] Webserver started at http://127.3.240.194:34827/ using document root <none> and password file <none>
I20260810 11:37:49.599836 12371 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:49.600150 12371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:49.600503 12371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:49.603657 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "df6e97f0a32942338bd8e73053a6be8b"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "3d13002ed77680291466ac9250405be7"
server_key_iv: "64c0faaff3586ca701db4c0bab2df575"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.604465 12371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "df6e97f0a32942338bd8e73053a6be8b"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "3d13002ed77680291466ac9250405be7"
server_key_iv: "64c0faaff3586ca701db4c0bab2df575"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.610283 12371 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:37:49.613988 12389 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.615495 12371 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 11:37:49.615685 12371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "df6e97f0a32942338bd8e73053a6be8b"
format_stamp: "Formatted at 2026-08-10 11:37:49 on dist-test-slave-drl0"
server_key: "3d13002ed77680291466ac9250405be7"
server_key_iv: "64c0faaff3586ca701db4c0bab2df575"
server_key_version: "encryptionkey@0"
I20260810 11:37:49.615904 12371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:49.629776 12371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:49.634727 12371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:49.635109 12371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:49.636109 12371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:49.637531 12371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:49.637820 12371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.638131 12371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:49.638263 12371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:49.678588 12371 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38883
I20260810 11:37:49.678642 12553 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38883 every 8 connection(s)
I20260810 11:37:49.680466 12371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:49.681119 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12371
I20260810 11:37:49.681314 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:49.681981 4035 external_mini_cluster.cc:1428] Setting key 17392a04fd5caa033e4c86b87a6a71cd
Aug 10 11:37:49 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361869, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:49.707791 12557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:49.683482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:35905 (local address 127.3.240.254:34593)
0810 11:37:49.683965 (+ 483us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:49.683972 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:49.685198 (+ 1226us) server_negotiation.cc:407] Connection header received
0810 11:37:49.688297 (+ 3099us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:49.688301 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:49.688366 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:49.688470 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:49.690712 (+ 2242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.691527 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.694121 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.694549 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.694658 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:49.699798 (+ 5140us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:49.699805 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:49.699808 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:49.699855 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:49.703758 (+ 3903us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:49.704755 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:49.704762 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:49.704766 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:49.704922 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:49.705428 (+ 506us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:49.705434 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:49.705439 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:49.705787 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:49.705961 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:49.706854 (+ 893us) server_negotiation.cc:299] Negotiation successful
0810 11:37:49.707230 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":259,"threads_started":1}
I20260810 11:37:49.708976 12558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:49.684135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34593 (local address 127.3.240.194:35905)
0810 11:37:49.685012 (+ 877us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:49.685071 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:49.687958 (+ 2887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:49.688761 (+ 803us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:49.688777 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:49.689606 (+ 829us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:49.690484 (+ 878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:49.690516 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.691706 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.691716 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:49.693944 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:49.693964 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:49.694800 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:49.694810 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:49.695204 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:49.696146 (+ 942us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:49.696170 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:49.699398 (+ 3228us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:49.703994 (+ 4596us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:49.704005 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:49.704026 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:49.704604 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:49.705055 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:49.705065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:49.705069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:49.705307 (+ 238us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:49.705930 (+ 623us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:49.705941 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:49.706388 (+ 447us) client_negotiation.cc:769] Sending connection context
0810 11:37:49.706727 (+ 339us) client_negotiation.cc:240] Negotiation successful
0810 11:37:49.707640 (+ 913us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":564,"thread_start_us":325,"threads_started":1}
I20260810 11:37:49.710819 12554 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34593
I20260810 11:37:49.711263 12554 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260810 11:37:49.712155 12554 heartbeater.cc:511] Master 127.3.240.254:34593 requested a full tablet report, sending...
I20260810 11:37:49.713958 12083 ts_manager.cc:194] Registered new tserver with Master: df6e97f0a32942338bd8e73053a6be8b (127.3.240.194:38883)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
I20260810 11:37:49.715119 12083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:35905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:37:49.748409 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44179
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:50.103529 12562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:50.103827 12562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:50.104002 12562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:50.111431 12562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:50.111547 12562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:50.111732 12562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:50.121713 12562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44179
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:34593
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:50.123323 12562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:50.124681 12562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:50.139108 12568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:50.139150 12567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:50.139690 12570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:50.139946 12562 server_base.cc:1034] running on GCE node
I20260810 11:37:50.140542 12562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:50.141422 12562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:50.142668 12562 hybrid_clock.cc:648] HybridClock initialized: now 1786361870142629 us; error 51 us; skew 500 ppm
I20260810 11:37:50.143138 12562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.147665 12562 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:37:50.149236 12562 webserver.cc:460] Webserver started at http://127.3.240.195:32865/ using document root <none> and password file <none>
I20260810 11:37:50.149946 12562 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:50.150090 12562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:50.150413 12562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:50.153635 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "65ce9a878acd4246b65476b03328694c"
format_stamp: "Formatted at 2026-08-10 11:37:50 on dist-test-slave-drl0"
server_key: "7f8a7a3f00a20613e8bfaf21cd3f5330"
server_key_iv: "140cfafd79cbf22c3d0efd761c4b87a7"
server_key_version: "encryptionkey@0"
I20260810 11:37:50.154445 12562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "65ce9a878acd4246b65476b03328694c"
format_stamp: "Formatted at 2026-08-10 11:37:50 on dist-test-slave-drl0"
server_key: "7f8a7a3f00a20613e8bfaf21cd3f5330"
server_key_iv: "140cfafd79cbf22c3d0efd761c4b87a7"
server_key_version: "encryptionkey@0"
I20260810 11:37:50.160903 12562 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260810 11:37:50.165136 12362 heartbeater.cc:503] Master 127.3.240.254:34593 was elected leader, sending a full tablet report...
I20260810 11:37:50.165917 12582 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:50.167465 12562 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:37:50.167634 12562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "65ce9a878acd4246b65476b03328694c"
format_stamp: "Formatted at 2026-08-10 11:37:50 on dist-test-slave-drl0"
server_key: "7f8a7a3f00a20613e8bfaf21cd3f5330"
server_key_iv: "140cfafd79cbf22c3d0efd761c4b87a7"
server_key_version: "encryptionkey@0"
I20260810 11:37:50.167835 12562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:50.182137 12562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:50.186595 12562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:50.186873 12562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:50.187948 12562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:50.189962 12562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:50.190157 12562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:50.190354 12562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:50.190424 12562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:50.230531 12562 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34743
I20260810 11:37:50.230620 12738 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34743 every 8 connection(s)
I20260810 11:37:50.231925 12562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:50.236832 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12562
I20260810 11:37:50.237030 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:50.237618 4035 external_mini_cluster.cc:1428] Setting key 55a050152a882c39c295850be715791a
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:37:50.257401 12743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.234675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:59253 (local address 127.3.240.254:34593)
0810 11:37:50.235184 (+ 509us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.235189 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.236658 (+ 1469us) server_negotiation.cc:407] Connection header received
0810 11:37:50.239163 (+ 2505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.239168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.239308 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.239488 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.242077 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.242875 (+ 798us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.244692 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.245194 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.245317 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.251019 (+ 5702us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.251026 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.251031 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.251078 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.254356 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.255126 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.255130 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.255133 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.255261 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.255770 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.255783 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.255785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.256068 (+ 283us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.256206 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.256936 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.257136 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":395,"thread_start_us":295,"threads_started":1}
I20260810 11:37:50.258415 12744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.235291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:34593 (local address 127.3.240.195:59253)
0810 11:37:50.236397 (+ 1106us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:50.236460 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:50.238783 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:50.239972 (+ 1189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:50.240003 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:50.240865 (+ 862us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:50.241843 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:50.241875 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.243197 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.243203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:50.244507 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:50.244522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.245482 (+ 960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.245489 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:50.245770 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.246732 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:50.246790 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:50.250708 (+ 3918us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:50.254500 (+ 3792us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:50.254507 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:50.254520 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:50.255027 (+ 507us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:50.255398 (+ 371us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:50.255401 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:50.255404 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:50.255619 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:50.256207 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:50.256213 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:50.256555 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 11:37:50.256801 (+ 246us) client_negotiation.cc:240] Negotiation successful
0810 11:37:50.257214 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":388,"threads_started":1}
I20260810 11:37:50.259966 12740 heartbeater.cc:347] Connected to a master server at 127.3.240.254:34593
I20260810 11:37:50.260298 12740 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:50.261174 12740 heartbeater.cc:511] Master 127.3.240.254:34593 requested a full tablet report, sending...
I20260810 11:37:50.262784 12083 ts_manager.cc:194] Registered new tserver with Master: 65ce9a878acd4246b65476b03328694c (127.3.240.195:34743)
I20260810 11:37:50.263623 12083 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:59253
I20260810 11:37:50.266884 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/test-admin.keytab.
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.300800 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260810 11:37:50.306839 12751 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:50.303908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40922 (local address 127.3.240.193:32931)
0810 11:37:50.304829 (+ 921us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.304838 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.304882 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:37:50.304955 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.304962 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.305206 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.305369 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.306439 (+ 1070us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40922: BlockingRecv error: recv got EOF from 127.0.0.1:40922 (error 108)
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":460,"threads_started":1}
W20260810 11:37:50.306900 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc' of type FILE
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.308913 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.310446 4035 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:50.329015 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.312307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40930 (local address 127.3.240.193:32931)
0810 11:37:50.312494 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.312499 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.312570 (+ 71us) server_negotiation.cc:407] Connection header received
0810 11:37:50.312869 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.312875 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.312970 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.313179 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.314387 (+ 1208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.315442 (+ 1055us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.316679 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.317240 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.317352 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.322482 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.322494 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.322509 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.322561 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.325636 (+ 3075us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.326308 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.326315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.326318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.326453 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.327019 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.327035 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.327038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.327449 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.327700 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.328258 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.328645 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 11:37:50.335736 12751 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:50.333135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40944 (local address 127.3.240.193:32931)
0810 11:37:50.333469 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.333477 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.333827 (+ 350us) server_negotiation.cc:407] Connection header received
0810 11:37:50.334073 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.334081 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.334176 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.334305 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.335431 (+ 1126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40944: BlockingRecv error: recv got EOF from 127.0.0.1:40944 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
W20260810 11:37:50.336491 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc' of type FILE
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.338644 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.340634 4035 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:50.357347 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.342414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40956 (local address 127.3.240.193:32931)
0810 11:37:50.342625 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.342629 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.342786 (+ 157us) server_negotiation.cc:407] Connection header received
0810 11:37:50.342921 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.342925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.343051 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.343140 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.344132 (+ 992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.344846 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.346359 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.346807 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.347004 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.351704 (+ 4700us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.351717 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.351722 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.351759 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.354717 (+ 2958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.355420 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.355425 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.355427 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.355551 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.355996 (+ 445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.356000 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.356003 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.356295 (+ 292us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.356513 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.356846 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.357075 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260810 11:37:50.362569 12751 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:37:50.360630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.3.240.193:32931)
0810 11:37:50.360864 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.360870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.361108 (+ 238us) server_negotiation.cc:407] Connection header received
0810 11:37:50.361291 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.361295 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.361370 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.361457 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.362343 (+ 886us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40972: BlockingRecv error: recv got EOF from 127.0.0.1:40972 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 11:37:50.362854 4035 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786361809851056-4035-0/krb5kdc/krb5cc' of type FILE
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.364801 4035 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:50.366439 4035 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:37:50.381927 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.368078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40980 (local address 127.3.240.193:32931)
0810 11:37:50.368302 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.368306 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.368373 (+ 67us) server_negotiation.cc:407] Connection header received
0810 11:37:50.368549 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.368553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.368609 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.368692 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.369654 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.370383 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.371803 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.372342 (+ 539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.372497 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.376115 (+ 3618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.376120 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.376123 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.376157 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.379010 (+ 2853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.379747 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.379753 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.379756 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.380110 (+ 354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.380511 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.380515 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.380518 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.380805 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.381067 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.381367 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.381635 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260810 11:37:50.383771 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12167
I20260810 11:37:50.396943 12751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.384614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40996 (local address 127.3.240.193:32931)
0810 11:37:50.384840 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.384848 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.385081 (+ 233us) server_negotiation.cc:407] Connection header received
0810 11:37:50.385276 (+ 195us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.385280 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.385336 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.385461 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.386497 (+ 1036us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.387430 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.388775 (+ 1345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.389192 (+ 417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.389300 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.391433 (+ 2133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.391437 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.391441 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.391473 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.394375 (+ 2902us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.395097 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.395101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.395103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.395237 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.395677 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.395680 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.395682 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.395929 (+ 247us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.396126 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.396492 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.396662 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:37:50.397547 12350 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:50.501156 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12167
I20260810 11:37:50.527256 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12371
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:37:50.545116 12767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.528186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32932 (local address 127.3.240.194:38883)
0810 11:37:50.528765 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.528772 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.528796 (+ 24us) server_negotiation.cc:407] Connection header received
0810 11:37:50.528859 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.528865 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.529079 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.529212 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.530325 (+ 1113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.531577 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.532852 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.533515 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.533627 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.537760 (+ 4133us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.537771 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.537785 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.537816 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.541512 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.542181 (+ 669us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.542188 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.542191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.542316 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.542804 (+ 488us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.542812 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.542815 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.543423 (+ 608us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.543826 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.544440 (+ 614us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.544748 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":423,"thread_start_us":310,"threads_started":1}
I20260810 11:37:50.546669 12546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:50.635870 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12371
I20260810 11:37:50.662905 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12562
Aug 10 11:37:50 dist-test-slave-drl0 krb5kdc[12018](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361870, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:37:50.680469 12770 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:50.664192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57626 (local address 127.3.240.195:34743)
0810 11:37:50.664923 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:50.664936 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:50.664995 (+ 59us) server_negotiation.cc:407] Connection header received
0810 11:37:50.665079 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:50.665086 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:50.665308 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:50.665450 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:50.666770 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.668033 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.669427 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:50.670098 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:50.670252 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:50.674409 (+ 4157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:50.674421 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:50.674436 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:50.674474 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:50.677379 (+ 2905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.678074 (+ 695us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.678082 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.678084 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.678210 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:50.678656 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:50.678660 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:50.678662 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:50.679219 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:50.679560 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:50.679870 (+ 310us) server_negotiation.cc:299] Negotiation successful
0810 11:37:50.680148 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":531,"thread_start_us":347,"threads_started":1}
I20260810 11:37:50.682397 12734 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:50.772137 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12562
I20260810 11:37:50.799028 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12035
I20260810 11:37:50.800344 12125 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:50.908879 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12035
2026-08-10T11:37:50Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4604 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:37:51 dist-test-slave-drl0 krb5kdc[12776](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:37:51 dist-test-slave-drl0 krb5kdc[12776](info): set up 2 sockets
Aug 10 11:37:51 dist-test-slave-drl0 krb5kdc[12776](info): commencing operation
krb5kdc: starting...
W20260810 11:37:52.265285 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.004s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:37:52 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:37:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:37:52Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
I20260810 11:37:52.543515 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38543
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38543
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:52.893415 12795 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:52.893689 12795 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:52.893817 12795 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:52.901047 12795 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:52.901149 12795 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:52.901238 12795 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:52.901325 12795 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:52.901401 12795 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:52.912612 12795 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38543
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38543
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:52.914058 12795 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:52.915413 12795 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:52.929632 12801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:52.932261 12806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:52.932747 12802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:52.933583 12795 server_base.cc:1034] running on GCE node
I20260810 11:37:52.934623 12795 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:52.936553 12795 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:52.937822 12795 hybrid_clock.cc:648] HybridClock initialized: now 1786361872937764 us; error 43 us; skew 500 ppm
I20260810 11:37:52.938282 12795 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:52 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:52.945924 12795 init.cc:377] Logged in from keytab as oryx/127.3.240.254@KRBTEST.COM (short username oryx)
I20260810 11:37:52.947885 12795 webserver.cc:460] Webserver started at http://127.3.240.254:46099/ using document root <none> and password file <none>
I20260810 11:37:52.948623 12795 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:52.948752 12795 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:52.949076 12795 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:52.952162 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "57926610f9004c339176098dcb8abf42"
format_stamp: "Formatted at 2026-08-10 11:37:52 on dist-test-slave-drl0"
server_key: "679d27587a42b0df9b9af2d2f5636409"
server_key_iv: "461f7660a4ebf7986e9f8d4bbadbcafd"
server_key_version: "encryptionkey@0"
I20260810 11:37:52.952963 12795 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "57926610f9004c339176098dcb8abf42"
format_stamp: "Formatted at 2026-08-10 11:37:52 on dist-test-slave-drl0"
server_key: "679d27587a42b0df9b9af2d2f5636409"
server_key_iv: "461f7660a4ebf7986e9f8d4bbadbcafd"
server_key_version: "encryptionkey@0"
I20260810 11:37:52.958132 12795 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:37:52.961897 12818 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:52.964064 12795 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 11:37:52.964356 12795 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "57926610f9004c339176098dcb8abf42"
format_stamp: "Formatted at 2026-08-10 11:37:52 on dist-test-slave-drl0"
server_key: "679d27587a42b0df9b9af2d2f5636409"
server_key_iv: "461f7660a4ebf7986e9f8d4bbadbcafd"
server_key_version: "encryptionkey@0"
I20260810 11:37:52.964630 12795 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:52.981189 12795 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:52.986141 12795 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:52.986452 12795 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:53.007448 12795 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38543
I20260810 11:37:53.007494 12890 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38543 every 8 connection(s)
I20260810 11:37:53.008994 12795 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:37:53.011289 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12795
I20260810 11:37:53.011528 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:37:53.012264 4035 external_mini_cluster.cc:1428] Setting key 4db70d7250689af5b1b0d8f8df494e23
I20260810 11:37:53.013458 12892 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 11:37:53 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:53.028864 12892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Bootstrap starting.
I20260810 11:37:53.033309 12892 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:53.034565 12892 log.cc:824] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:53.037557 12892 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: No bootstrap required, opened a new log
I20260810 11:37:53.038226 12897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:53.014371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.3.240.254:38543)
0810 11:37:53.015301 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:53.015312 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:53.015358 (+ 46us) server_negotiation.cc:407] Connection header received
0810 11:37:53.017055 (+ 1697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:53.017077 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:53.017483 (+ 406us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:53.017856 (+ 373us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:53.019817 (+ 1961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.021144 (+ 1327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.022910 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.023636 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.023783 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:53.028988 (+ 5205us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:53.029000 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:53.029018 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:53.029063 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:53.033657 (+ 4594us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:53.034376 (+ 719us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:53.034385 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:53.034395 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:53.034613 (+ 218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:53.035346 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:53.035358 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:53.035361 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:53.035943 (+ 582us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:53.036346 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:53.036801 (+ 455us) server_negotiation.cc:299] Negotiation successful
0810 11:37:53.037218 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":415,"threads_started":1}
I20260810 11:37:53.045799 12892 raft_consensus.cc:348] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } }
I20260810 11:37:53.046144 12892 raft_consensus.cc:374] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:53.046250 12892 raft_consensus.cc:729] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 57926610f9004c339176098dcb8abf42, State: Initialized, Role: FOLLOWER
I20260810 11:37:53.046824 12892 consensus_queue.cc:260] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } }
I20260810 11:37:53.047097 12892 raft_consensus.cc:388] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:37:53.047274 12892 raft_consensus.cc:482] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:37:53.047472 12892 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:53.049594 12892 raft_consensus.cc:504] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } }
I20260810 11:37:53.050100 12892 leader_election.cc:302] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 57926610f9004c339176098dcb8abf42; no voters:
I20260810 11:37:53.050765 12892 leader_election.cc:288] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:37:53.050920 12901 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:37:53.051807 12901 raft_consensus.cc:686] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 LEADER]: Becoming Leader. State: Replica: 57926610f9004c339176098dcb8abf42, State: Running, Role: LEADER
I20260810 11:37:53.052452 12901 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } }
I20260810 11:37:53.053325 12892 sys_catalog.cc:564] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:53.056243 12904 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57926610f9004c339176098dcb8abf42. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } }
I20260810 11:37:53.056352 12903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } }
I20260810 11:37:53.056875 12903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:53.056802 12904 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:53.058758 12908 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:37:53.063411 12908 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:37:53.072630 12908 catalog_manager.cc:1347] Generated new cluster ID: dee69dbb8d9348d097481bbd6c939eb8
I20260810 11:37:53.072742 12908 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:37:53.099324 12908 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:37:53.100770 12908 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:37:53.109789 12908 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Generated new TSK 0
I20260810 11:37:53.110984 12908 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
I20260810 11:37:53.188385 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:53.576651 12931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:53.576922 12931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:53.577018 12931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:53.584327 12931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:53.584447 12931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:53.584560 12931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:53.595815 12931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:53.597081 12931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:53.598434 12931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:53.612334 12943 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:53.612334 12938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:53.612586 12939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:53.614216 12931 server_base.cc:1034] running on GCE node
I20260810 11:37:53.614832 12931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:53.615887 12931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:53.617183 12931 hybrid_clock.cc:648] HybridClock initialized: now 1786361873617149 us; error 63 us; skew 500 ppm
I20260810 11:37:53.617558 12931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:53 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361873, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:53.620860 12931 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260810 11:37:53.622747 12931 webserver.cc:460] Webserver started at http://127.3.240.193:34273/ using document root <none> and password file <none>
I20260810 11:37:53.623502 12931 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:53.623610 12931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:53.623873 12931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:53.626685 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "6160c92c581a455798298753a797bbfb"
format_stamp: "Formatted at 2026-08-10 11:37:53 on dist-test-slave-drl0"
server_key: "c422865ca4687c1fa6f8a14f33d8fd7a"
server_key_iv: "044bde9dedefdd6a83780844dc37270f"
server_key_version: "encryptionkey@0"
I20260810 11:37:53.627498 12931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "6160c92c581a455798298753a797bbfb"
format_stamp: "Formatted at 2026-08-10 11:37:53 on dist-test-slave-drl0"
server_key: "c422865ca4687c1fa6f8a14f33d8fd7a"
server_key_iv: "044bde9dedefdd6a83780844dc37270f"
server_key_version: "encryptionkey@0"
I20260810 11:37:53.633376 12931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:37:53.638541 12952 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:53.640441 12931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:53.640616 12931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "6160c92c581a455798298753a797bbfb"
format_stamp: "Formatted at 2026-08-10 11:37:53 on dist-test-slave-drl0"
server_key: "c422865ca4687c1fa6f8a14f33d8fd7a"
server_key_iv: "044bde9dedefdd6a83780844dc37270f"
server_key_version: "encryptionkey@0"
I20260810 11:37:53.640995 12931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:53.668687 12931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:53.673374 12931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:53.673635 12931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:53.674604 12931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:53.676168 12931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:53.676247 12931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:53.676335 12931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:53.676443 12931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:53.720901 12931 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38681
I20260810 11:37:53.720973 13118 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38681 every 8 connection(s)
I20260810 11:37:53.722395 12931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:53.723528 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 12931
I20260810 11:37:53.723757 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:37:53.724846 4035 external_mini_cluster.cc:1428] Setting key ee08ac768e4256358cd28b6519f2d750
Aug 10 11:37:53 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361873, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:53.750053 13124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:53.726105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56363 (local address 127.3.240.254:38543)
0810 11:37:53.726607 (+ 502us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:53.726613 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:53.727752 (+ 1139us) server_negotiation.cc:407] Connection header received
0810 11:37:53.729790 (+ 2038us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:53.729796 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:53.729881 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:53.730015 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:53.731898 (+ 1883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.733129 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.734806 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.735309 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.735473 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:53.740807 (+ 5334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:53.740819 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:53.740827 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:53.740899 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:53.745682 (+ 4783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:53.746645 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:53.746654 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:53.746660 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:53.746833 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:53.747478 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:53.747486 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:53.747492 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:53.747898 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:53.748059 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:53.749296 (+ 1237us) server_negotiation.cc:299] Negotiation successful
0810 11:37:53.749563 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":375,"thread_start_us":250,"threads_started":1}
I20260810 11:37:53.751368 13125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:53.726727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:56363)
0810 11:37:53.727595 (+ 868us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:53.727642 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:53.729500 (+ 1858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:53.730259 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:53.730268 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:53.730974 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:53.731700 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:53.731722 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.733322 (+ 1600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.733327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:53.734631 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:53.734644 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:53.735561 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:53.735571 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:53.735803 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:53.737090 (+ 1287us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:53.737113 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:53.740301 (+ 3188us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:53.745874 (+ 5573us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:53.745882 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:53.745895 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:53.746495 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:53.747126 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:53.747129 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:53.747132 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:53.747331 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:53.748175 (+ 844us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:53.748186 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:53.748764 (+ 578us) client_negotiation.cc:769] Sending connection context
0810 11:37:53.749120 (+ 356us) client_negotiation.cc:240] Negotiation successful
0810 11:37:53.749692 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":387,"threads_started":1}
I20260810 11:37:53.753562 13120 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:53.753978 13120 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:53.755070 13120 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:37:53.757720 12842 ts_manager.cc:194] Registered new tserver with Master: 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:53.759552 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:56363
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
I20260810 11:37:53.795385 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:54.162405 13128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:54.162668 13128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:54.162762 13128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:54.170060 13128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:54.170173 13128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:54.170290 13128 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:54.180336 13128 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:54.181797 13128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:54.183427 13128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:54.196817 13135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:54.196780 13141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:54.196841 13134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:54.198835 13128 server_base.cc:1034] running on GCE node
I20260810 11:37:54.199847 13128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:54.201169 13128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:54.203076 13128 hybrid_clock.cc:648] HybridClock initialized: now 1786361874202991 us; error 63 us; skew 500 ppm
I20260810 11:37:54.203410 13128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:54 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361874, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:54.206848 13128 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260810 11:37:54.208747 13128 webserver.cc:460] Webserver started at http://127.3.240.194:45423/ using document root <none> and password file <none>
I20260810 11:37:54.209522 13128 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:54.209630 13128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:54.209947 13128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:54.213111 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "9c8090f25aa7434e856fb7949607deb7"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "c1a3144d9c523b46bd581c663ac019cf"
server_key_iv: "6c707f2930c4c9d8e01da3cf709a8f01"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.214344 13128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "9c8090f25aa7434e856fb7949607deb7"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "c1a3144d9c523b46bd581c663ac019cf"
server_key_iv: "6c707f2930c4c9d8e01da3cf709a8f01"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.220849 13128 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 11:37:54.224946 13150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.226547 13128 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260810 11:37:54.226722 13128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "9c8090f25aa7434e856fb7949607deb7"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "c1a3144d9c523b46bd581c663ac019cf"
server_key_iv: "6c707f2930c4c9d8e01da3cf709a8f01"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.226971 13128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:54.250818 13128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:54.255672 13128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:54.256145 13128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:54.257268 13128 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:54.258829 13128 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:54.258996 13128 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.259162 13128 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:54.259243 13128 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.300606 13128 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:43305
I20260810 11:37:54.300663 13321 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:43305 every 8 connection(s)
I20260810 11:37:54.301982 13128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:54.306643 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 13128
I20260810 11:37:54.306860 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:37:54.307381 4035 external_mini_cluster.cc:1428] Setting key eb893e67b678116c9772364c10ea33e5
Aug 10 11:37:54 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361874, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:54.325682 13324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:54.304953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58783 (local address 127.3.240.254:38543)
0810 11:37:54.305451 (+ 498us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:54.305455 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:54.306435 (+ 980us) server_negotiation.cc:407] Connection header received
0810 11:37:54.308650 (+ 2215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:54.308654 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:54.308719 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:54.308831 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:54.310862 (+ 2031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.311900 (+ 1038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.313978 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.314445 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.314614 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:54.318588 (+ 3974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:54.318594 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:54.318597 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:54.318634 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:54.321931 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:54.322996 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:54.323002 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:54.323006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:54.323145 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:54.323748 (+ 603us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:54.323752 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:54.323755 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:54.324057 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:54.324175 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:54.325212 (+ 1037us) server_negotiation.cc:299] Negotiation successful
0810 11:37:54.325394 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":262,"threads_started":1}
I20260810 11:37:54.327327 13325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:54.305513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:58783)
0810 11:37:54.306264 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:54.306312 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:54.308329 (+ 2017us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:54.309082 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:54.309093 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:54.309778 (+ 685us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:54.310664 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:54.310686 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.312233 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.312239 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:54.313777 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:54.313796 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.314720 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.314728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:54.315084 (+ 356us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:54.316023 (+ 939us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:54.316052 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:54.318317 (+ 2265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:54.322108 (+ 3791us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:54.322120 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:54.322140 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:54.322777 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:54.323286 (+ 509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:54.323295 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:54.323302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:54.323566 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:54.324215 (+ 649us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:54.324225 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:54.324737 (+ 512us) client_negotiation.cc:769] Sending connection context
0810 11:37:54.325119 (+ 382us) client_negotiation.cc:240] Negotiation successful
0810 11:37:54.325725 (+ 606us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":306,"threads_started":1}
I20260810 11:37:54.328953 13322 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:54.329279 13322 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:54.330909 13322 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
I20260810 11:37:54.333351 12842 ts_manager.cc:194] Registered new tserver with Master: 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305)
I20260810 11:37:54.334340 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:58783
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260810 11:37:54.375164 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:37:54.741261 13329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:54.741552 13329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:54.741654 13329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:54.750070 13329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:54.750185 13329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:54.750298 13329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:54.761893 13329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38543
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:54.762907 13120 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
I20260810 11:37:54.763471 13329 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:54.765178 13329 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:54.780650 13339 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:54.781020 13335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:54.781068 13337 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:54.782581 13329 server_base.cc:1034] running on GCE node
I20260810 11:37:54.783357 13329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:54.784451 13329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:54.785718 13329 hybrid_clock.cc:648] HybridClock initialized: now 1786361874785676 us; error 61 us; skew 500 ppm
I20260810 11:37:54.786074 13329 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:54 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361874, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:54.789326 13329 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260810 11:37:54.790813 13329 webserver.cc:460] Webserver started at http://127.3.240.195:33097/ using document root <none> and password file <none>
I20260810 11:37:54.791536 13329 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:54.791639 13329 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:54.792001 13329 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:54.794888 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "d812df4cd66145dd8e713375f6ad2e3d"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "2e8b4e8295faa1e8c87dbfb34e73fa3f"
server_key_iv: "56ea9a3d34373403531242160c666f11"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.795660 13329 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "d812df4cd66145dd8e713375f6ad2e3d"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "2e8b4e8295faa1e8c87dbfb34e73fa3f"
server_key_iv: "56ea9a3d34373403531242160c666f11"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.800827 13329 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260810 11:37:54.804390 13348 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.805871 13329 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:54.806039 13329 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "d812df4cd66145dd8e713375f6ad2e3d"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "2e8b4e8295faa1e8c87dbfb34e73fa3f"
server_key_iv: "56ea9a3d34373403531242160c666f11"
server_key_version: "encryptionkey@0"
I20260810 11:37:54.806232 13329 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:54.831650 13329 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:54.836860 13329 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:54.837270 13329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:54.838543 13329 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:54.840451 13329 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:54.840546 13329 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.840641 13329 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:54.840752 13329 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:54.881230 13329 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33373
I20260810 11:37:54.881248 13504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33373 every 8 connection(s)
I20260810 11:37:54.882664 13329 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:54.883641 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 13329
I20260810 11:37:54.883831 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:37:54.884409 4035 external_mini_cluster.cc:1428] Setting key 04a164a8bfd08bc2e25795996459d015
Aug 10 11:37:54 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361874, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:54.906689 13509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:54.886077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:39831 (local address 127.3.240.254:38543)
0810 11:37:54.886551 (+ 474us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:54.886556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:54.888876 (+ 2320us) server_negotiation.cc:407] Connection header received
0810 11:37:54.890978 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:54.890984 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:54.891093 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:54.891228 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:54.892838 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.893623 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.895232 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.895631 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.895777 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:54.899990 (+ 4213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:54.900001 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:54.900010 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:54.900080 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:54.903311 (+ 3231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:54.904061 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:54.904066 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:54.904069 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:54.904195 (+ 126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:54.905027 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:54.905031 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:54.905033 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:54.905322 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:54.905456 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:54.906223 (+ 767us) server_negotiation.cc:299] Negotiation successful
0810 11:37:54.906405 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":252,"threads_started":1}
I20260810 11:37:54.907631 13510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:54.887295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:39831)
0810 11:37:54.888646 (+ 1351us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:54.888728 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:54.890651 (+ 1923us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:54.891562 (+ 911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:54.891572 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:54.892085 (+ 513us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:54.892651 (+ 566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:54.892669 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.893799 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.893802 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:54.895089 (+ 1287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:54.895102 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:54.895869 (+ 767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:54.895876 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:54.896117 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:54.897006 (+ 889us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:54.897029 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:54.899619 (+ 2590us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:54.903459 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:54.903466 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:54.903477 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:54.903911 (+ 434us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:54.904378 (+ 467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:54.904384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:54.904387 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:54.904805 (+ 418us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:54.905499 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:54.905506 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:54.905824 (+ 318us) client_negotiation.cc:769] Sending connection context
0810 11:37:54.906067 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 11:37:54.906467 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":870,"thread_start_us":489,"threads_started":1}
I20260810 11:37:54.909825 13505 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:54.910115 13505 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:54.910820 13505 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
I20260810 11:37:54.912472 12842 ts_manager.cc:194] Registered new tserver with Master: d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373)
I20260810 11:37:54.913447 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:39831
I20260810 11:37:54.914595 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.253.keytab.
Entry for principal oryx/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.253.keytab.
I20260810 11:37:54.987300 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34331
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.253.keytab
--principal=oryx/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38543,127.3.240.253:34331
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:55.339246 13322 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
W20260810 11:37:55.367807 13515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:55.368167 13515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:55.368328 13515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:55.376530 13515 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:55.376688 13515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:55.376811 13515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:55.376910 13515 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:55.376995 13515 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:55.387184 13515 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38543,127.3.240.253:34331
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.253.keytab
--principal=oryx/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34331
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
W20260810 11:37:55.387614 13515 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.3.240.254:38543,127.3.240.253:34331'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260810 11:37:55.389303 13515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:55.391052 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:55.404419 13524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:55.404467 13525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:55.404508 13527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:55.405658 13515 server_base.cc:1034] running on GCE node
I20260810 11:37:55.406508 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:55.408087 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:55.409402 13515 hybrid_clock.cc:648] HybridClock initialized: now 1786361875409308 us; error 95 us; skew 500 ppm
I20260810 11:37:55.410107 13515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361875, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:55.415495 13515 init.cc:377] Logged in from keytab as oryx/127.3.240.253@KRBTEST.COM (short username oryx)
I20260810 11:37:55.421924 13515 webserver.cc:460] Webserver started at http://127.3.240.253:42673/ using document root <none> and password file <none>
I20260810 11:37:55.423225 13515 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:55.423481 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:55.424083 13515 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:55.428721 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/instance:
uuid: "ee426fdc32f54c42b732c4d50632a67e"
format_stamp: "Formatted at 2026-08-10 11:37:55 on dist-test-slave-drl0"
server_key: "c6b1eda4508556dad385653c46a412c1"
server_key_iv: "dac1f7cd6476a39083e656df2433b3c3"
server_key_version: "encryptionkey@0"
I20260810 11:37:55.429610 13515 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal/instance:
uuid: "ee426fdc32f54c42b732c4d50632a67e"
format_stamp: "Formatted at 2026-08-10 11:37:55 on dist-test-slave-drl0"
server_key: "c6b1eda4508556dad385653c46a412c1"
server_key_iv: "dac1f7cd6476a39083e656df2433b3c3"
server_key_version: "encryptionkey@0"
I20260810 11:37:55.436067 13515 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 11:37:55.440008 13537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:55.441912 13515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:55.442229 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "ee426fdc32f54c42b732c4d50632a67e"
format_stamp: "Formatted at 2026-08-10 11:37:55 on dist-test-slave-drl0"
server_key: "c6b1eda4508556dad385653c46a412c1"
server_key_iv: "dac1f7cd6476a39083e656df2433b3c3"
server_key_version: "encryptionkey@0"
I20260810 11:37:55.442526 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:55.470929 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:55.476222 13515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:55.476526 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:55.498138 13515 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:34331
I20260810 11:37:55.498183 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:34331 every 8 connection(s)
I20260810 11:37:55.499527 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
I20260810 11:37:55.503464 13601 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:55.507229 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 13515
I20260810 11:37:55.511848 13601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361875, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:55.533972 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.515003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51064 (local address 127.3.240.254:38543)
0810 11:37:55.516231 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.516236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.516254 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:55.516318 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.516323 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.516419 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.516534 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.518040 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.519062 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.520920 (+ 1858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.521343 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.521445 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.526648 (+ 5203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.526657 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.526664 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.526711 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.530008 (+ 3297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.530963 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.530968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.530971 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.531133 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.531858 (+ 725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.531869 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.531873 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.532522 (+ 649us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.532660 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.533302 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.533559 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1105,"thread_start_us":998,"threads_started":1}
I20260810 11:37:55.534825 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.514894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51064)
0810 11:37:55.515767 (+ 873us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.515819 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.516011 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.516625 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.516629 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.517488 (+ 859us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.517842 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.517855 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.519204 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.519208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.520745 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.520765 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.521597 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.521606 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.521812 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.522451 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.522480 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.526372 (+ 3892us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.530196 (+ 3824us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.530209 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.530225 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.530797 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.531314 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.531320 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.531324 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.531563 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.532689 (+ 1126us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.532697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.532901 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.533125 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.533526 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"mutex_wait_us":80,"spinlock_wait_cycles":38144,"thread_start_us":319,"threads_started":1}
I20260810 11:37:55.534869 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.509650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35314 (local address 127.3.240.253:34331)
0810 11:37:55.510643 (+ 993us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.510655 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.510696 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:37:55.512550 (+ 1854us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.512571 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.512953 (+ 382us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.513332 (+ 379us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.514841 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.516879 (+ 2038us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.518549 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.519499 (+ 950us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.519635 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.525620 (+ 5985us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.525641 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.525664 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.525734 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.529711 (+ 3977us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.530541 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.530574 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.530577 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.530770 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.531314 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.531320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.531324 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.532022 (+ 698us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.532261 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.532909 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.533391 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":528,"threads_started":1}
I20260810 11:37:55.537918 13601 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361875, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260810 11:37:55.553253 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.539064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35320)
0810 11:37:55.539232 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.539248 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.539384 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.539655 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.539659 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.540024 (+ 365us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.540142 (+ 118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.540150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.541397 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.541405 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.542487 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.542500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.543280 (+ 780us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.543285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.543465 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.543974 (+ 509us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.543991 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.546109 (+ 2118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.550111 (+ 4002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.550118 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.550122 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.550802 (+ 680us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.551303 (+ 501us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.551308 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.551311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.551492 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.552225 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.552229 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.552402 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.552548 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.552844 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69}
I20260810 11:37:55.553375 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.539243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35320 (local address 127.3.240.253:34331)
0810 11:37:55.539397 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.539402 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.539420 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:37:55.539478 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.539482 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.539543 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.539642 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.540413 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.541237 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.542636 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.543138 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.543275 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.546315 (+ 3040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.546320 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.546324 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.546443 (+ 119us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.549899 (+ 3456us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.550998 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.551003 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.551006 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.551127 (+ 121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.551678 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.551690 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.551693 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.552050 (+ 357us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.552158 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.552669 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.553101 (+ 432us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":155,"server-negotiator.queue_time_us":64}
I20260810 11:37:55.558842 13601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Bootstrap starting.
I20260810 11:37:55.562666 13601 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:55.564019 13601 log.cc:824] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:55.567154 13601 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: No bootstrap required, opened a new log
I20260810 11:37:55.575702 13601 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } }
I20260810 11:37:55.576068 13601 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:55.576181 13601 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee426fdc32f54c42b732c4d50632a67e, State: Initialized, Role: FOLLOWER
I20260810 11:37:55.576818 13601 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } }
I20260810 11:37:55.577661 13613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } } }
I20260810 11:37:55.578104 13613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:37:55.578769 13601 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:55.580247 13613 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:55.580608 13613 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } }
I20260810 11:37:55.582306 13613 leader_election.cc:288] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543)
I20260810 11:37:55.583264 12857 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee426fdc32f54c42b732c4d50632a67e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "57926610f9004c339176098dcb8abf42" is_pre_election: true
I20260810 11:37:55.584499 13542 leader_election.cc:302] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: ee426fdc32f54c42b732c4d50632a67e; no voters: 57926610f9004c339176098dcb8abf42
I20260810 11:37:55.585006 13613 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260810 11:37:55.591704 13631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:37:55.591913 13631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:37:55.595062 13515 master_runner.cc:428] Detected that this master ee426fdc32f54c42b732c4d50632a67e is joining an existing cluster
I20260810 11:37:55.595429 13515 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.240.253:34331
WARNING: no policy specified for oryx/127.3.240.252@KRBTEST.COM; defaulting to no policy
I20260810 11:37:55.633261 13640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.602502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35336)
0810 11:37:55.613041 (+ 10539us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.613064 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.613534 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.614095 (+ 561us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.614103 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.614956 (+ 853us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.615395 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.615412 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.617007 (+ 1595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.617014 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.618914 (+ 1900us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.618955 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.620473 (+ 1518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.620482 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.620750 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.621645 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.621674 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.623143 (+ 1469us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.629622 (+ 6479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.629628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.629632 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.630210 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.630815 (+ 605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.630819 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.630823 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.631067 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.632021 (+ 954us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.632025 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.632176 (+ 151us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.632366 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.632699 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10341,"thread_start_us":10650,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.252@KRBTEST.COM" created.
I20260810 11:37:55.633315 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.603109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.3.240.253:34331)
0810 11:37:55.603379 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.603389 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.613152 (+ 9763us) server_negotiation.cc:407] Connection header received
0810 11:37:55.613674 (+ 522us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.613683 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.613764 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.613833 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.615546 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.616760 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.619086 (+ 2326us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.620014 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.620127 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.623368 (+ 3241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.623377 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.623384 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.623423 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.629482 (+ 6059us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.630347 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.630353 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.630356 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.630673 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.631197 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.631202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.631205 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.631783 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.631944 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.632515 (+ 571us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.632856 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":81,"server-negotiator.queue_time_us":43}
I20260810 11:37:55.643275 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.603311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.3.240.254:38543)
0810 11:37:55.603512 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.603520 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.623630 (+ 20110us) server_negotiation.cc:407] Connection header received
0810 11:37:55.623963 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.623969 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.624092 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.624197 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.625902 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.627093 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.629333 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.630028 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.630209 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.633526 (+ 3317us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.633535 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.633543 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.633590 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.639906 (+ 6316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.640848 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.640854 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.640858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.641058 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.641497 (+ 439us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.641503 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.641507 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.641906 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.642074 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.642598 (+ 524us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.642882 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:37:55.643473 13643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.602547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51076)
0810 11:37:55.623497 (+ 20950us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.623514 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.623676 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.624530 (+ 854us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.624536 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.625290 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.625686 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.625703 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.627251 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.627257 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.629114 (+ 1857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.629132 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.630274 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.630283 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.630517 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.632035 (+ 1518us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.632052 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.633220 (+ 1168us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.640096 (+ 6876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.640104 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.640111 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.640701 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.641188 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.641196 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.641202 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.641372 (+ 170us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.642046 (+ 674us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.642054 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.642286 (+ 232us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.642470 (+ 184us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.642849 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20795,"thread_start_us":20579,"threads_started":1}
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.252.keytab.
Entry for principal oryx/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.252.keytab.
I20260810 11:37:55.675820 13604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.651742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35346 (local address 127.3.240.253:34331)
0810 11:37:55.651950 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.651956 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.655687 (+ 3731us) server_negotiation.cc:407] Connection header received
0810 11:37:55.656100 (+ 413us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.656109 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.656254 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.656365 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.657878 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.659169 (+ 1291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.661702 (+ 2533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.662570 (+ 868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.662802 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.668221 (+ 5419us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.668229 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.668235 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.668275 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.672535 (+ 4260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.673367 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.673373 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.673377 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.673543 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.674075 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.674083 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.674086 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.674473 (+ 387us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.674637 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.675173 (+ 536us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.675422 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 11:37:55.675953 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39553
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.252.keytab
--principal=oryx/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:55.675876 13646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.651592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35346)
0810 11:37:55.655536 (+ 3944us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.655597 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.655849 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.656437 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.656445 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.657377 (+ 932us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.657685 (+ 308us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.657701 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.659324 (+ 1623us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.659330 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.661341 (+ 2011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.661378 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.662767 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.662775 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.663040 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.663836 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.663861 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.667952 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.672709 (+ 4757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.672720 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.672726 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.673239 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.673683 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.673687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.673689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.673939 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.674617 (+ 678us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.674625 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.674822 (+ 197us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.675080 (+ 258us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.675423 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3697,"thread_start_us":3537,"threads_started":1}
I20260810 11:37:55.678853 12842 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.240.253:34331: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e"
member_type: NON_VOTER
last_known_addr {
host: "127.3.240.253"
port: 34331
}
attrs {
promote: true
}
}
dest_uuid: "57926610f9004c339176098dcb8abf42"
cas_config_opid_index: -1
I20260810 11:37:55.680328 12842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } }
I20260810 11:37:55.692283 13652 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee426fdc32f54c42b732c4d50632a67e (127.3.240.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.693467 13572 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:55.697381 13572 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260810 11:37:55.698292 13648 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:37:55.701965 13652 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.240.253:34331
I20260810 11:37:55.703037 13515 master.cc:537] Master@127.3.240.253:34331 shutting down...
I20260810 11:37:55.704211 13652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.704546 13652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:55.739428 13572 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER ee426fdc32f54c42b732c4d50632a67e (127.3.240.253) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.744853 13664 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.745329 13664 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:55.745776 13613 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57926610f9004c339176098dcb8abf42. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.745854 13660 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.746351 13660 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:55.746647 13613 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:55.753842 13515 raft_consensus.cc:2229] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Raft consensus shutting down.
I20260810 11:37:55.754366 13515 raft_consensus.cc:2258] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Raft consensus is shut down!
I20260810 11:37:55.754653 13515 tablet_replica.cc:330] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: stopping tablet replica
I20260810 11:37:55.764676 13515 master.cc:552] Master@127.3.240.253:34331 shutdown complete.
I20260810 11:37:55.764830 13515 master_runner.cc:305] Clearing existing system tablet
I20260810 11:37:55.765350 13515 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:55.765497 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:55.768402 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 11:37:55.770064 13669 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:55.770350 13515 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:55.770706 13515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 11:37:55.770836 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "ee426fdc32f54c42b732c4d50632a67e"
format_stamp: "Formatted at 2026-08-10 11:37:55 on dist-test-slave-drl0"
server_key: "c6b1eda4508556dad385653c46a412c1"
server_key_iv: "dac1f7cd6476a39083e656df2433b3c3"
server_key_version: "encryptionkey@0"
I20260810 11:37:55.775045 13515 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:55.781231 13515 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 11:37:55.781461 13515 log.cc:1196] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260810 11:37:55.781973 13515 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Deleting consensus metadata
I20260810 11:37:55.782243 13515 master_runner.cc:315] Copying system tablet from 127.3.240.254:38543
I20260810 11:37:55.785336 13515 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: tablet copy: Beginning tablet copy session from remote peer at address 127.3.240.254:38543
I20260810 11:37:55.801429 13608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.786300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51078 (local address 127.3.240.254:38543)
0810 11:37:55.786561 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:55.786570 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:55.787125 (+ 555us) server_negotiation.cc:407] Connection header received
0810 11:37:55.787342 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:55.787348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:55.787425 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:55.787533 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:55.788353 (+ 820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.789241 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.791529 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.792002 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.792154 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.794499 (+ 2345us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:55.794505 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:55.794517 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:55.794548 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:55.798367 (+ 3819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.799337 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.799344 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.799348 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.799497 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:55.799911 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:55.799916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:55.799920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:55.800205 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:55.800372 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:55.800767 (+ 395us) server_negotiation.cc:299] Negotiation successful
0810 11:37:55.800996 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 11:37:55.801440 13678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:55.786266 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51078)
0810 11:37:55.787014 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:55.787030 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:55.787207 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:55.787567 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:55.787571 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:55.787992 (+ 421us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:55.788202 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.788211 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.789491 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.789500 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.791354 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:55.791367 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:55.792187 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:55.792193 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:55.792399 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:55.793090 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:55.793107 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:55.794252 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:55.798525 (+ 4273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.798531 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.798536 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.799211 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.799627 (+ 416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:55.799633 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:55.799637 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:55.799785 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:55.800339 (+ 554us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:55.800344 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:55.800509 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 11:37:55.800654 (+ 145us) client_negotiation.cc:240] Negotiation successful
0810 11:37:55.800997 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":463,"threads_started":1}
I20260810 11:37:55.802613 12874 tablet_copy_service.cc:140] P 57926610f9004c339176098dcb8abf42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer ee426fdc32f54c42b732c4d50632a67e ({username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:51078)
I20260810 11:37:55.803030 12874 tablet_copy_service.cc:161] P 57926610f9004c339176098dcb8abf42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer ee426fdc32f54c42b732c4d50632a67e at {username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:51078: session id = ee426fdc32f54c42b732c4d50632a67e-00000000000000000000000000000000
I20260810 11:37:55.806416 12874 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 11:37:55.809381 13515 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 11:37:55.816009 13515 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: tablet copy: Starting download of 0 data blocks...
I20260810 11:37:55.816509 13515 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: tablet copy: Starting download of 1 WAL segments...
I20260810 11:37:55.820258 13515 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 11:37:55.824204 12874 tablet_copy_service.cc:342] P 57926610f9004c339176098dcb8abf42: Request end of tablet copy session ee426fdc32f54c42b732c4d50632a67e-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.240.253@KRBTEST.COM'} at 127.0.0.1:51078
I20260810 11:37:55.824585 12874 tablet_copy_service.cc:433] P 57926610f9004c339176098dcb8abf42: ending tablet copy session ee426fdc32f54c42b732c4d50632a67e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer ee426fdc32f54c42b732c4d50632a67e
I20260810 11:37:55.838002 13515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:55.841331 13679 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:55.842072 13680 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:55.846719 13515 server_base.cc:1034] running on GCE node
W20260810 11:37:55.847419 13683 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:55.847832 13515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:55.848546 13515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:55.850018 13515 hybrid_clock.cc:648] HybridClock initialized: now 1786361875849898 us; error 42 us; skew 500 ppm
I20260810 11:37:55.850404 13515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:55 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361875, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:55.852975 13515 init.cc:377] Logged in from keytab as oryx/127.3.240.253@KRBTEST.COM (short username oryx)
I20260810 11:37:55.860682 13515 webserver.cc:460] Webserver started at http://127.3.240.253:45029/ using document root <none> and password file <none>
I20260810 11:37:55.861143 13515 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:55.861243 13515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:55.863826 13515 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260810 11:37:55.866411 13693 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:55.866904 13515 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260810 11:37:55.867084 13515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "ee426fdc32f54c42b732c4d50632a67e"
format_stamp: "Formatted at 2026-08-10 11:37:55 on dist-test-slave-drl0"
server_key: "c6b1eda4508556dad385653c46a412c1"
server_key_iv: "dac1f7cd6476a39083e656df2433b3c3"
server_key_version: "encryptionkey@0"
I20260810 11:37:55.867261 13515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:55.898214 13515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:55.903297 13515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:55.916290 13505 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
I20260810 11:37:55.956172 13515 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:34331
I20260810 11:37:55.956195 13758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:34331 every 8 connection(s)
I20260810 11:37:55.957285 13515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
I20260810 11:37:55.965149 13760 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:37:55.973280 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Bootstrap starting.
I20260810 11:37:55.987964 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:37:55.988587 13760 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: Bootstrap complete.
I20260810 11:37:55.989770 13760 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } }
I20260810 11:37:55.990046 13760 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: ee426fdc32f54c42b732c4d50632a67e, State: Initialized, Role: LEARNER
I20260810 11:37:55.990392 13760 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } }
I20260810 11:37:55.992364 13760 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:55.992091 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:55.992733 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:56.004685 13788 catalog_manager.cc:1259] Loaded cluster ID: dee69dbb8d9348d097481bbd6c939eb8
I20260810 11:37:56.004796 13788 catalog_manager.cc:1552] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: loading cluster ID for follower catalog manager: success
I20260810 11:37:56.008834 13788 catalog_manager.cc:1574] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: acquiring CA information for follower catalog manager: success
I20260810 11:37:56.012097 13788 catalog_manager.cc:1602] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260810 11:37:56.232236 13653 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:56.232590 13653 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:56.232784 13653 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:56.241516 13653 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:37:56.241681 13653 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:56.241802 13653 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:56.241982 13653 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:37:56.242106 13653 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:37:56.260314 13653 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.252.keytab
--principal=oryx/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39553
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:56.262322 13653 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:56.264149 13653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:56.283674 13795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:56.284044 13796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:56.284993 13800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:56.285543 13653 server_base.cc:1034] running on GCE node
I20260810 11:37:56.286815 13653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:56.290110 13653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:56.291471 13653 hybrid_clock.cc:648] HybridClock initialized: now 1786361876291368 us; error 81 us; skew 500 ppm
I20260810 11:37:56.292136 13653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361876, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:56.297695 13653 init.cc:377] Logged in from keytab as oryx/127.3.240.252@KRBTEST.COM (short username oryx)
I20260810 11:37:56.300346 13653 webserver.cc:460] Webserver started at http://127.3.240.252:43493/ using document root <none> and password file <none>
I20260810 11:37:56.301599 13653 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:56.301790 13653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:56.302574 13653 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:37:56.303663 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:56.304934 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:56.308341 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/instance:
uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
format_stamp: "Formatted at 2026-08-10 11:37:56 on dist-test-slave-drl0"
server_key: "20759420e2cded8959f44d20bc062740"
server_key_iv: "3273dad0932d13d052300bbf274111ee"
server_key_version: "encryptionkey@0"
I20260810 11:37:56.309860 13653 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal/instance:
uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
format_stamp: "Formatted at 2026-08-10 11:37:56 on dist-test-slave-drl0"
server_key: "20759420e2cded8959f44d20bc062740"
server_key_iv: "3273dad0932d13d052300bbf274111ee"
server_key_version: "encryptionkey@0"
I20260810 11:37:56.319340 13653 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.008s sys 0.003s
I20260810 11:37:56.321600 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.303834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35362)
0810 11:37:56.311503 (+ 7669us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.311529 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.311793 (+ 264us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.312634 (+ 841us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.312641 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.312675 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:56.313099 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.313115 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.316120 (+ 3005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.316127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.318687 (+ 2560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.318705 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.320927 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.320932 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.321093 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.321134 (+ 41us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.321206 (+ 72us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.321318 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7479,"thread_start_us":7304,"threads_started":1}
I20260810 11:37:56.321957 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.304317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35362 (local address 127.3.240.253:34331)
0810 11:37:56.305166 (+ 849us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.305173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.311650 (+ 6477us) server_negotiation.cc:407] Connection header received
0810 11:37:56.311991 (+ 341us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.312000 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.312113 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.312225 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:56.313417 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.315828 (+ 2411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.318900 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.320734 (+ 1834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.320883 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.321169 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.321255 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.321422 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":491,"threads_started":1}
I20260810 11:37:56.323146 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57926610f9004c339176098dcb8abf42. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: NON_VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: true } } }
I20260810 11:37:56.323518 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:56.324139 13812 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: attempting to promote NON_VOTER ee426fdc32f54c42b732c4d50632a67e to VOTER
I20260810 11:37:56.325132 13819 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:56.325403 13812 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } }
I20260810 11:37:56.327122 13653 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260810 11:37:56.327371 13653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
format_stamp: "Formatted at 2026-08-10 11:37:56 on dist-test-slave-drl0"
server_key: "20759420e2cded8959f44d20bc062740"
server_key_iv: "3273dad0932d13d052300bbf274111ee"
server_key_version: "encryptionkey@0"
I20260810 11:37:56.327728 13727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 LEARNER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260810 11:37:56.327611 13653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:56.328346 13807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260810 11:37:56.333232 13809 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee426fdc32f54c42b732c4d50632a67e (127.3.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } }
I20260810 11:37:56.333999 13727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, ee426fdc32f54c42b732c4d50632a67e (127.3.240.253) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } }
I20260810 11:37:56.338200 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } }
I20260810 11:37:56.338526 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:56.338572 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } }
I20260810 11:37:56.338860 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:56.338847 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } }
I20260810 11:37:56.339159 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:37:56.355844 13653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:56.361460 13653 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:56.361804 13653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:56.394413 13653 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:39553
I20260810 11:37:56.394420 13894 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:39553 every 8 connection(s)
I20260810 11:37:56.396277 13653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
I20260810 11:37:56.402367 13895 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:56.405697 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 13653
I20260810 11:37:56.412715 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361876, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:56.433027 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.408307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37386 (local address 127.3.240.252:39553)
0810 11:37:56.409657 (+ 1350us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.409669 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.409727 (+ 58us) server_negotiation.cc:407] Connection header received
0810 11:37:56.412344 (+ 2617us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.412369 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.412794 (+ 425us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.413255 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.415165 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.417111 (+ 1946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.418513 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.419369 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.419592 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.424754 (+ 5162us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.424762 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.424773 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.424808 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.428448 (+ 3640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.429145 (+ 697us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.429152 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.429158 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.429321 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.429824 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.429829 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.429832 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.430511 (+ 679us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.430883 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.431480 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.431879 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1047,"thread_start_us":772,"threads_started":1}
I20260810 11:37:56.436184 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.415721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51082 (local address 127.3.240.254:38543)
0810 11:37:56.416479 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.416485 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.420051 (+ 3566us) server_negotiation.cc:407] Connection header received
0810 11:37:56.420308 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.420318 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.420386 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.420528 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.421998 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.423076 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.424709 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.425221 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.425362 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.429198 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.429203 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.429206 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.429237 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.432869 (+ 3632us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.433820 (+ 951us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.433826 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.433829 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.433980 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.434509 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.434515 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.434519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.434810 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.435008 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.435508 (+ 500us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.435731 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":601,"thread_start_us":420,"threads_started":1}
I20260810 11:37:56.436188 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.415837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51082)
0810 11:37:56.419874 (+ 4037us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.419929 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.420111 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.420607 (+ 496us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.420615 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.421428 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.421789 (+ 361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.421799 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.423237 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.423241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.424548 (+ 1307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.424561 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.425448 (+ 887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.425457 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.425690 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.426497 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.426515 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.428997 (+ 2482us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.433019 (+ 4022us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.433031 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.433039 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.433693 (+ 654us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.434099 (+ 406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.434104 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.434108 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.434392 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.434977 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.434984 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.435190 (+ 206us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.435420 (+ 230us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.435731 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3681,"thread_start_us":3444,"threads_started":1}
I20260810 11:37:56.437839 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361876, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260810 11:37:56.457150 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.438894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35364 (local address 127.3.240.253:34331)
0810 11:37:56.439100 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.439104 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.439389 (+ 285us) server_negotiation.cc:407] Connection header received
0810 11:37:56.439527 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.439532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.439620 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.439709 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.440700 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.441803 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.443748 (+ 1945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.444571 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.444763 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.449628 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.449635 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.449639 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.449686 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.453494 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.454491 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.454496 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.454499 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.454749 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.455280 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.455286 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.455288 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.455696 (+ 408us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.455838 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.456554 (+ 716us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.456795 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 11:37:56.457247 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.438812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35364)
0810 11:37:56.439197 (+ 385us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.439224 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.439393 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.439803 (+ 410us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.439808 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.440303 (+ 495us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.440487 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.440496 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.442266 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.442270 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.443529 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.443545 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.444895 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.444906 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.445101 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.445919 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.445941 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.449258 (+ 3317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.453762 (+ 4504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.453767 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.453771 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.454255 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.454883 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.454886 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.454889 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.455113 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.455914 (+ 801us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.455920 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.456209 (+ 289us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.456399 (+ 190us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.456780 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":95}
I20260810 11:37:56.458493 13895 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361876, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260810 11:37:56.473510 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.459272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.3.240.252:39553)
0810 11:37:56.459523 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.459530 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.459626 (+ 96us) server_negotiation.cc:407] Connection header received
0810 11:37:56.459804 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.459808 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.459874 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.459977 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.461043 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.461899 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.463277 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.463705 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.463842 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.467580 (+ 3738us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.467585 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.467588 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.467619 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.470520 (+ 2901us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.471334 (+ 814us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.471339 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.471343 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.471465 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.471907 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.471912 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.471915 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.472296 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.472458 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.472830 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.473166 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":103}
I20260810 11:37:56.473529 13903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.459216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.0.0.1:37398)
0810 11:37:56.459523 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.459551 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.459687 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.459997 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.460001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.460559 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.460874 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.460884 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.462041 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.462047 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.463141 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.463153 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.463886 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.463892 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.464130 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.464801 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.464819 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.467394 (+ 2575us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.470701 (+ 3307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.470707 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.470710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.471212 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.471575 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.471580 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.471584 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.471781 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.472420 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.472425 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.472601 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.472740 (+ 139us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.473015 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260810 11:37:56.479205 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Bootstrap starting.
I20260810 11:37:56.483170 13895 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:56.484509 13895 log.cc:824] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:56.487641 13895 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: No bootstrap required, opened a new log
I20260810 11:37:56.496805 13895 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } }
I20260810 11:37:56.497174 13895 raft_consensus.cc:374] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:56.497375 13895 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 970e8a6cd0a840ab99fabcf63b58ee1f, State: Initialized, Role: FOLLOWER
I20260810 11:37:56.497952 13895 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } }
I20260810 11:37:56.499037 13905 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } } }
I20260810 11:37:56.499536 13905 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:37:56.500142 13895 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: configured and running, proceeding with master startup.
W20260810 11:37:56.514955 13921 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:37:56.515151 13921 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:37:56.519977 13653 master_runner.cc:428] Detected that this master 970e8a6cd0a840ab99fabcf63b58ee1f is joining an existing cluster
I20260810 11:37:56.520121 13653 master_runner.cc:432] Initiating AddMaster RPC to add 127.3.240.252:39553
I20260810 11:37:56.539623 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12931
I20260810 11:37:56.556159 13814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.525556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35380 (local address 127.3.240.253:34331)
0810 11:37:56.525755 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.525760 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.531340 (+ 5580us) server_negotiation.cc:407] Connection header received
0810 11:37:56.531717 (+ 377us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.531723 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.531814 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.531904 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.535539 (+ 3635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.536736 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.539211 (+ 2475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.539982 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.540088 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.544369 (+ 4281us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.544377 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.544381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.544425 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.549874 (+ 5449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.551241 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.551248 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.551251 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.551520 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.552195 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.552201 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.552204 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.552689 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.552858 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.555068 (+ 2210us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.555387 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:37:56.556572 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.525442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51092 (local address 127.3.240.254:38543)
0810 11:37:56.525706 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.525713 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.527626 (+ 1913us) server_negotiation.cc:407] Connection header received
0810 11:37:56.527995 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.528002 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.528112 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.528257 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.530870 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.532281 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.536421 (+ 4140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.537165 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.537271 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.544860 (+ 7589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.544868 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.544873 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.544928 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.551092 (+ 6164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.552187 (+ 1095us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.552211 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.552215 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.552489 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.553082 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.553087 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.553090 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.553976 (+ 886us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.554176 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.555505 (+ 1329us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.555992 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 11:37:56.557644 13932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.525762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35380)
0810 11:37:56.531194 (+ 5432us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.531249 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.531460 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.532148 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.532152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.533892 (+ 1740us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.535361 (+ 1469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.535378 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.536984 (+ 1606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.536989 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.539035 (+ 2046us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.539051 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.540449 (+ 1398us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.540456 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.540639 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.541832 (+ 1193us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.541880 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.544035 (+ 2155us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.550030 (+ 5995us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.550042 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.550049 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.551052 (+ 1003us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.551700 (+ 648us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.551705 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.551709 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.552032 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.552936 (+ 904us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.552941 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.553631 (+ 690us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.553781 (+ 150us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.554297 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5284,"thread_start_us":1082,"threads_started":1}
I20260810 11:37:56.561792 13930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.525734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51092)
0810 11:37:56.527470 (+ 1736us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.527501 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.527788 (+ 287us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.528317 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.528326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.529775 (+ 1449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.530625 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.530648 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.532661 (+ 2013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.532670 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.536211 (+ 3541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.536226 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.537649 (+ 1423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.537656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.541241 (+ 3585us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.542610 (+ 1369us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.542643 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.544435 (+ 1792us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.551388 (+ 6953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.551395 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.551400 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.552021 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.552689 (+ 668us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.552693 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.552697 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.552944 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.554371 (+ 1427us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.554376 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.554986 (+ 610us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.556782 (+ 1796us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.560669 (+ 3887us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1457,"thread_start_us":4997,"threads_started":1}
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.193@KRBTEST.COM
I20260810 11:37:56.573549 13934 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.531877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.0.0.1:37414)
0810 11:37:56.534434 (+ 2557us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.534468 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.534675 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.535044 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.535048 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.536147 (+ 1099us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.536741 (+ 594us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.536756 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.538603 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.538608 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.541509 (+ 2901us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.541532 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.544853 (+ 3321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.544863 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.545313 (+ 450us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.554837 (+ 9524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.554863 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.556886 (+ 2023us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.563993 (+ 7107us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.564000 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.564005 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.564921 (+ 916us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.565508 (+ 587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.565512 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.565515 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.565738 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.566564 (+ 826us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.566569 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.566857 (+ 288us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.567089 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.572615 (+ 5526us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2265,"thread_start_us":1793,"threads_started":1}
I20260810 11:37:56.574918 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.531575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37414 (local address 127.3.240.252:39553)
0810 11:37:56.531770 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.531776 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.534612 (+ 2836us) server_negotiation.cc:407] Connection header received
0810 11:37:56.534796 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.534802 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.534881 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.535027 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.536911 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.538287 (+ 1376us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.541710 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.544140 (+ 2430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.544279 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.557084 (+ 12805us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.557089 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.557095 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.557132 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.563818 (+ 6686us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.565116 (+ 1298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.565121 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.565123 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.565381 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.565875 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.565882 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.565887 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.566438 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.573855 (+ 7417us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.574152 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.574477 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 11:37:56.578447 13939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.540854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60378 (local address 127.3.240.193:38681)
0810 11:37:56.546673 (+ 5819us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.546684 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.546763 (+ 79us) server_negotiation.cc:407] Connection header received
0810 11:37:56.546880 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.546890 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.547328 (+ 438us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.550375 (+ 3047us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.550865 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.553270 (+ 2405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.556457 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.558109 (+ 1652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.565313 (+ 7204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.567754 (+ 2441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.567772 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.567786 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.567821 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.572547 (+ 4726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.573421 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.573432 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.573435 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.573698 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.574330 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.574337 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.574341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.574765 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.576939 (+ 2174us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.577355 (+ 416us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.577707 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5479,"thread_start_us":4961,"threads_started":1}
I20260810 11:37:56.580266 13111 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260810 11:37:56.604013 13900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.577789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37416 (local address 127.3.240.252:39553)
0810 11:37:56.578000 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.578006 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.578025 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:37:56.578114 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.578120 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.578235 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.579865 (+ 1630us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.580229 (+ 364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.582540 (+ 2311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.585229 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.586743 (+ 1514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.587858 (+ 1115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.593069 (+ 5211us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.593085 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.593092 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.593142 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.599072 (+ 5930us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.600626 (+ 1554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.600632 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.600636 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.600861 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.601512 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.601518 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.601522 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.602063 (+ 541us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.602227 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.603131 (+ 904us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.603434 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:37:56.604040 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.577017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.0.0.1:37416)
0810 11:37:56.577424 (+ 407us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.577444 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.577549 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.578489 (+ 940us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.578494 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.579282 (+ 788us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.579698 (+ 416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.579713 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.582713 (+ 3000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.582719 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.584863 (+ 2144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.584880 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.586980 (+ 2100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.586990 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.587283 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.588532 (+ 1249us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.588555 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.592737 (+ 4182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.599232 (+ 6495us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.599238 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.599244 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.600438 (+ 1194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.601064 (+ 626us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.601070 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.601073 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.601338 (+ 265us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.602356 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.602362 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.602743 (+ 381us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.602962 (+ 219us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.603356 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":269}
I20260810 11:37:56.607237 12842 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.3.240.252:39553: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
member_type: NON_VOTER
last_known_addr {
host: "127.3.240.252"
port: 39553
}
attrs {
promote: true
}
}
dest_uuid: "57926610f9004c339176098dcb8abf42"
cas_config_opid_index: 6
I20260810 11:37:56.608672 12842 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } }
I20260810 11:37:56.612459 13859 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:56.612519 13727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 11:37:56.613483 13807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260810 11:37:56.616676 13807 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.619792 13727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, NON_VOTER 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252) added. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.622481 13807 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.3.240.252:39553
I20260810 11:37:56.623334 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.623842 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:56.624343 13809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.624670 13809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:56.625834 13764 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.626291 13764 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:37:56.635836 13653 master.cc:537] Master@127.3.240.252:39553 shutting down...
W20260810 11:37:56.635809 13833 tablet.cc:2369] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:56.636148 13859 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260810 11:37:56.637374 13807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 11:37:56.641647 12824 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260810 11:37:56.655121 12824 consensus_peers.cc:487] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 -> Peer 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252:39553): Couldn't send request to peer 970e8a6cd0a840ab99fabcf63b58ee1f. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260810 11:37:56.657424 13653 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:37:56.657895 13653 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:37:56.658169 13653 tablet_replica.cc:330] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: stopping tablet replica
I20260810 11:37:56.667476 13653 master.cc:552] Master@127.3.240.252:39553 shutdown complete.
I20260810 11:37:56.667582 13653 master_runner.cc:305] Clearing existing system tablet
I20260810 11:37:56.668148 13653 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:56.668305 13653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:56.672063 13653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:37:56.674018 13944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:56.674484 13653 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:56.674819 13653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 11:37:56.674983 13653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
format_stamp: "Formatted at 2026-08-10 11:37:56 on dist-test-slave-drl0"
server_key: "20759420e2cded8959f44d20bc062740"
server_key_iv: "3273dad0932d13d052300bbf274111ee"
server_key_version: "encryptionkey@0"
I20260810 11:37:56.680968 13653 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:37:56.688591 13653 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260810 11:37:56.688840 13653 log.cc:1196] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260810 11:37:56.689391 13653 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Deleting consensus metadata
I20260810 11:37:56.689715 13653 master_runner.cc:315] Copying system tablet from 127.3.240.254:38543
I20260810 11:37:56.692698 13653 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: tablet copy: Beginning tablet copy session from remote peer at address 127.3.240.254:38543
I20260810 11:37:56.703059 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12931
I20260810 11:37:56.713362 13901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.693838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51098 (local address 127.3.240.254:38543)
0810 11:37:56.694081 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:56.694088 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:56.695629 (+ 1541us) server_negotiation.cc:407] Connection header received
0810 11:37:56.695864 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:56.695871 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:56.695957 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:56.696075 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:56.697294 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.698655 (+ 1361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.701407 (+ 2752us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.702077 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.702221 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.705428 (+ 3207us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:56.705438 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:56.705445 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:56.705512 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:56.709805 (+ 4293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.710801 (+ 996us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.710806 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.710809 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.710997 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:56.711494 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:56.711501 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:56.711505 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:56.711872 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:56.712016 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:56.712620 (+ 604us) server_negotiation.cc:299] Negotiation successful
0810 11:37:56.712894 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 11:37:56.713397 13956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:56.693827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51098)
0810 11:37:56.695471 (+ 1644us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:56.695496 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:56.695680 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:56.696262 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:56.696268 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:56.696900 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:56.697148 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.697162 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.698820 (+ 1658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.698826 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.701225 (+ 2399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:56.701241 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:56.702237 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:56.702243 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:56.702439 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:56.703397 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:56.703422 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:56.704593 (+ 1171us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:56.710015 (+ 5422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.710021 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.710025 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.710638 (+ 613us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.711153 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:56.711158 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:56.711162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:56.711355 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:56.712102 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:56.712107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:56.712294 (+ 187us) client_negotiation.cc:769] Sending connection context
0810 11:37:56.712448 (+ 154us) client_negotiation.cc:240] Negotiation successful
0810 11:37:56.712778 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1490,"thread_start_us":1372,"threads_started":1}
I20260810 11:37:56.715804 12874 tablet_copy_service.cc:140] P 57926610f9004c339176098dcb8abf42: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 970e8a6cd0a840ab99fabcf63b58ee1f ({username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:51098)
I20260810 11:37:56.716178 12874 tablet_copy_service.cc:161] P 57926610f9004c339176098dcb8abf42: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 970e8a6cd0a840ab99fabcf63b58ee1f at {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:51098: session id = 970e8a6cd0a840ab99fabcf63b58ee1f-00000000000000000000000000000000
I20260810 11:37:56.719995 12874 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Tablet Copy: opened 0 blocks and 1 log segments
I20260810 11:37:56.722664 13653 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260810 11:37:56.730949 13653 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: tablet copy: Starting download of 0 data blocks...
I20260810 11:37:56.731297 13653 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: tablet copy: Starting download of 1 WAL segments...
I20260810 11:37:56.733906 13653 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260810 11:37:56.735281 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:38681
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:38543
--webserver_port=34273
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:56.738485 12874 tablet_copy_service.cc:342] P 57926610f9004c339176098dcb8abf42: Request end of tablet copy session 970e8a6cd0a840ab99fabcf63b58ee1f-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:51098
I20260810 11:37:56.738801 12874 tablet_copy_service.cc:433] P 57926610f9004c339176098dcb8abf42: ending tablet copy session 970e8a6cd0a840ab99fabcf63b58ee1f-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 970e8a6cd0a840ab99fabcf63b58ee1f
I20260810 11:37:56.752429 13653 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:56.755537 13960 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:56.756449 13962 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:56.763237 13964 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:56.768656 13653 server_base.cc:1034] running on GCE node
I20260810 11:37:56.769049 13653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:56.769758 13653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:56.770985 13653 hybrid_clock.cc:648] HybridClock initialized: now 1786361876770940 us; error 36 us; skew 500 ppm
I20260810 11:37:56.771235 13653 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:56 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361876, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:56.773386 13653 init.cc:377] Logged in from keytab as oryx/127.3.240.252@KRBTEST.COM (short username oryx)
I20260810 11:37:56.780238 13653 webserver.cc:460] Webserver started at http://127.3.240.252:34589/ using document root <none> and password file <none>
I20260810 11:37:56.780577 13653 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:56.780684 13653 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:56.784397 13653 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.000s sys 0.004s
I20260810 11:37:56.786620 13975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:56.787117 13653 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260810 11:37:56.787249 13653 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "970e8a6cd0a840ab99fabcf63b58ee1f"
format_stamp: "Formatted at 2026-08-10 11:37:56 on dist-test-slave-drl0"
server_key: "20759420e2cded8959f44d20bc062740"
server_key_iv: "3273dad0932d13d052300bbf274111ee"
server_key_version: "encryptionkey@0"
I20260810 11:37:56.787463 13653 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:56.807224 13653 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:56.811861 13653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:56.861598 13653 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:39553
I20260810 11:37:56.861621 14047 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:39553 every 8 connection(s)
I20260810 11:37:56.862556 13653 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
I20260810 11:37:56.869390 14051 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:37:56.874075 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Bootstrap starting.
I20260810 11:37:56.898342 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:37:56.898921 14051 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Bootstrap complete.
I20260810 11:37:56.900071 14051 raft_consensus.cc:348] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } }
I20260810 11:37:56.900377 14051 raft_consensus.cc:729] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 970e8a6cd0a840ab99fabcf63b58ee1f, State: Initialized, Role: LEARNER
I20260810 11:37:56.900635 14051 consensus_queue.cc:260] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } }
I20260810 11:37:56.903434 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:56.903779 14051 sys_catalog.cc:564] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:37:56.904043 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: LEARNER
I20260810 11:37:56.914763 14074 catalog_manager.cc:1259] Loaded cluster ID: dee69dbb8d9348d097481bbd6c939eb8
I20260810 11:37:56.914867 14074 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: loading cluster ID for follower catalog manager: success
I20260810 11:37:56.918555 14074 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: acquiring CA information for follower catalog manager: success
I20260810 11:37:56.921260 14074 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:37:57.061935 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.053066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37430 (local address 127.3.240.252:39553)
0810 11:37:57.053932 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:57.053939 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:57.053959 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:37:57.054038 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:57.054043 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:57.054137 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:57.054277 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:57.054838 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.056565 (+ 1727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.059100 (+ 2535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.060509 (+ 1409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.060681 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.060899 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:57.061358 (+ 459us) server_negotiation.cc:299] Negotiation successful
0810 11:37:57.061505 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":447,"threads_started":1}
I20260810 11:37:57.061971 13813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.052797 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.0.0.1:37430)
0810 11:37:57.053096 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:57.053118 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:57.053270 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:57.054326 (+ 1056us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:57.054341 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:57.054393 (+ 52us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:57.054627 (+ 234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.054639 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.056726 (+ 2087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.056732 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.058728 (+ 1996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.058748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.060892 (+ 2144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.060898 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.061153 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.061203 (+ 50us) client_negotiation.cc:769] Sending connection context
0810 11:37:57.061317 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 11:37:57.061497 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":164}
I20260810 11:37:57.063799 14012 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.7] Dedup: 1.7->[]
I20260810 11:37:57.064498 14058 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 57926610f9004c339176098dcb8abf42. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:57.064828 14058 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: LEARNER
W20260810 11:37:57.231266 13959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:57.231721 13959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:57.232003 13959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:57.242292 13959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:57.242482 13959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:57.242779 13959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:37:57.255764 13959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:38681
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=34273
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:57.257395 13959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:57.258847 13959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:57.276895 14082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:57.277395 14084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:57.277808 14086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:57.278801 13959 server_base.cc:1034] running on GCE node
I20260810 11:37:57.280098 13959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:57.281368 13959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:57.282718 13959 hybrid_clock.cc:648] HybridClock initialized: now 1786361877282588 us; error 55 us; skew 500 ppm
I20260810 11:37:57.283375 13959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:57 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361877, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:57.288676 13959 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260810 11:37:57.290750 13959 webserver.cc:460] Webserver started at http://127.3.240.193:34273/ using document root <none> and password file <none>
I20260810 11:37:57.291491 13959 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:57.291620 13959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:57.297405 13959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:37:57.301048 14097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:57.302605 13959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:37:57.302768 13959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "6160c92c581a455798298753a797bbfb"
format_stamp: "Formatted at 2026-08-10 11:37:53 on dist-test-slave-drl0"
server_key: "c422865ca4687c1fa6f8a14f33d8fd7a"
server_key_iv: "044bde9dedefdd6a83780844dc37270f"
server_key_version: "encryptionkey@0"
I20260810 11:37:57.304419 13959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:57.325206 13959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:57.332343 13959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:57.332890 13959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:57.335003 13959 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:57.337172 13959 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:57.337388 13959 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:57.337595 13959 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:57.337839 13959 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:57.420758 13959 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38681
I20260810 11:37:57.420977 14263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38681 every 8 connection(s)
I20260810 11:37:57.423058 13959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:37:57.426889 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 13959
I20260810 11:37:57.427321 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 13128
Aug 10 11:37:57 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.194@KRBTEST.COM
Aug 10 11:37:57 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361877, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
Aug 10 11:37:57 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361877, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260810 11:37:57.462396 14277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.429254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39124 (local address 127.3.240.194:43305)
0810 11:37:57.432720 (+ 3466us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:57.432727 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:57.432750 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:57.432808 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:57.432813 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:57.433010 (+ 197us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:57.433180 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:57.435292 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.437282 (+ 1990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.440101 (+ 2819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.441351 (+ 1250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.441503 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.450101 (+ 8598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:57.450122 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:57.450140 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:57.450220 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:57.455839 (+ 5619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.458194 (+ 2355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.458204 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.458207 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.458470 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.459395 (+ 925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.459400 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.459403 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.460022 (+ 619us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:57.460680 (+ 658us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:57.461567 (+ 887us) server_negotiation.cc:299] Negotiation successful
0810 11:37:57.461884 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3305,"thread_start_us":3079,"threads_started":1}
Aug 10 11:37:57 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361877, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:57.464303 13313 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:37:57.469872 14273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.428317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46805 (local address 127.3.240.253:34331)
0810 11:37:57.428954 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:57.428961 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:57.436894 (+ 7933us) server_negotiation.cc:407] Connection header received
0810 11:37:57.441577 (+ 4683us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:57.441586 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:57.441776 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:57.441889 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:57.447401 (+ 5512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.449279 (+ 1878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.452505 (+ 3226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.453242 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.453344 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.459915 (+ 6571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:57.459922 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:57.459928 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:57.459968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:57.464957 (+ 4989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.466259 (+ 1302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.466267 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.466274 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.466428 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.467588 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.467593 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.467596 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.468008 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:57.468170 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:57.469167 (+ 997us) server_negotiation.cc:299] Negotiation successful
0810 11:37:57.469392 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":390,"threads_started":1}
I20260810 11:37:57.471364 14076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.429444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51271 (local address 127.3.240.252:39553)
0810 11:37:57.429702 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:57.429711 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:57.436121 (+ 6410us) server_negotiation.cc:407] Connection header received
0810 11:37:57.443792 (+ 7671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:57.443799 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:57.443915 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:57.444004 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:57.446741 (+ 2737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.448086 (+ 1345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.451529 (+ 3443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.452442 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.452554 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.458911 (+ 6357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:57.458921 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:57.458991 (+ 70us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:57.459063 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:57.466021 (+ 6958us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.467910 (+ 1889us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.467916 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.467919 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.468101 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.468665 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.468671 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.468675 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.469060 (+ 385us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:57.469216 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:57.470621 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0810 11:37:57.470896 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260810 11:37:57.472883 14278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.428608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.3.240.193:51271)
0810 11:37:57.435935 (+ 7327us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:57.435999 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:57.443541 (+ 7542us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:57.444490 (+ 949us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:57.444498 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:57.445342 (+ 844us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:57.446421 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.446473 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.448335 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.448341 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.451319 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.451338 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.453063 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.453075 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.453468 (+ 393us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.454574 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:57.454599 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:57.458182 (+ 3583us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:57.467120 (+ 8938us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.467129 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.467137 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.467735 (+ 598us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.468253 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.468258 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.468263 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.468496 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.469274 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:57.469279 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:57.469543 (+ 264us) client_negotiation.cc:769] Sending connection context
0810 11:37:57.469714 (+ 171us) client_negotiation.cc:240] Negotiation successful
0810 11:37:57.470319 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6925,"mutex_wait_us":39,"spinlock_wait_cycles":16512,"thread_start_us":7674,"threads_started":1}
I20260810 11:37:57.472961 14279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.436889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:52429)
0810 11:37:57.446344 (+ 9455us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:57.446371 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:57.446544 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:57.451729 (+ 5185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:57.451737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:57.452601 (+ 864us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:57.453161 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.453176 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.455038 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.455044 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.456956 (+ 1912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.456969 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.457963 (+ 994us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.457970 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.458221 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.459065 (+ 844us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:57.459089 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:57.463875 (+ 4786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:57.469274 (+ 5399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.469279 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.469283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.469875 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.470532 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.470537 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.470541 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.470744 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.471704 (+ 960us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:57.471711 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:57.471880 (+ 169us) client_negotiation.cc:769] Sending connection context
0810 11:37:57.472049 (+ 169us) client_negotiation.cc:240] Negotiation successful
0810 11:37:57.472345 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9172,"thread_start_us":6618,"threads_started":1}
I20260810 11:37:57.473843 14280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.436986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52429 (local address 127.3.240.254:38543)
0810 11:37:57.451382 (+ 14396us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:57.451388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:57.451411 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:37:57.451473 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:57.451478 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:57.451554 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:57.453331 (+ 1777us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:57.453553 (+ 222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.454790 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.457154 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.457656 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.457889 (+ 233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.464153 (+ 6264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:57.464162 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:57.464166 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:57.464212 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:57.469106 (+ 4894us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.470216 (+ 1110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.470221 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.470224 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.470390 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:57.470903 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:57.470907 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:57.470910 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:57.471552 (+ 642us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:57.472772 (+ 1220us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:57.472936 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 11:37:57.473435 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14265,"thread_start_us":14156,"threads_started":1}
I20260810 11:37:57.474021 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:57.428608 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.3.240.193:46805)
0810 11:37:57.436792 (+ 8184us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:57.436810 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:57.441047 (+ 4237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:57.442871 (+ 1824us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:57.442889 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:57.445047 (+ 2158us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:57.447141 (+ 2094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.447158 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.449592 (+ 2434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.449600 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.452257 (+ 2657us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:57.452276 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:57.453742 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:57.453750 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:57.453925 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:57.454561 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:57.454638 (+ 77us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:57.459658 (+ 5020us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:57.465222 (+ 5564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.465235 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.465255 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.466109 (+ 854us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.466585 (+ 476us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:57.466592 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:57.466598 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:57.467430 (+ 832us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:57.468273 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:57.468282 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:57.468728 (+ 446us) client_negotiation.cc:769] Sending connection context
0810 11:37:57.470133 (+ 1405us) client_negotiation.cc:240] Negotiation successful
0810 11:37:57.470634 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7287,"mutex_wait_us":1470,"spinlock_wait_cycles":19840,"thread_start_us":5966,"threads_started":1}
I20260810 11:37:57.475486 14266 heartbeater.cc:347] Connected to a master server at 127.3.240.252:39553
I20260810 11:37:57.475622 14269 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34331
I20260810 11:37:57.475628 14264 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:57.475988 14264 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:57.475968 14269 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:57.475966 14266 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:57.477623 14266 heartbeater.cc:511] Master 127.3.240.252:39553 requested a full tablet report, sending...
I20260810 11:37:57.477636 14269 heartbeater.cc:511] Master 127.3.240.253:34331 requested a full tablet report, sending...
I20260810 11:37:57.477696 14264 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
I20260810 11:37:57.480406 13713 ts_manager.cc:194] Registered new tserver with Master: 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:57.480388 13996 ts_manager.cc:194] Registered new tserver with Master: 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:57.481490 12842 ts_manager.cc:194] Re-registered known tserver with Master: 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:57.482343 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:52429
I20260810 11:37:57.561931 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 13128
I20260810 11:37:57.573755 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: NON_VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: true } } }
I20260810 11:37:57.574213 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:57.575273 13807 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: attempting to promote NON_VOTER 970e8a6cd0a840ab99fabcf63b58ee1f to VOTER
I20260810 11:37:57.576109 13807 consensus_queue.cc:237] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } }
I20260810 11:37:57.578330 14012 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 LEARNER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 11:37:57.578347 13727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Refusing update from remote peer 57926610f9004c339176098dcb8abf42: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260810 11:37:57.579108 14283 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 11:37:57.579680 13807 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260810 11:37:57.582641 13807 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.585944 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.586283 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:57.586565 13807 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.585739 14012 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.587221 13807 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:57.589003 13727 raft_consensus.cc:2932] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.593994 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:43305
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:38543
--webserver_port=45423
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:57.599648 14289 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.600153 14289 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:37:57.601694 14284 sys_catalog.cc:455] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.602111 14284 sys_catalog.cc:458] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42 [sys.catalog]: This master's current role is: LEADER
I20260810 11:37:57.605635 14290 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "57926610f9004c339176098dcb8abf42" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:37:57.606042 14290 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
W20260810 11:37:57.973282 14292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:57.973716 14292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:57.973878 14292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:57.981217 14292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:57.981369 14292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:57.981912 14292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:37:57.993542 14292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:43305
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=45423
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:57.995138 14292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:57.996498 14292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:58.013931 14298 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:58.014438 14299 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:58.015146 14302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:58.019151 14292 server_base.cc:1034] running on GCE node
I20260810 11:37:58.020262 14292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:58.021478 14292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:58.022756 14292 hybrid_clock.cc:648] HybridClock initialized: now 1786361878022687 us; error 63 us; skew 500 ppm
I20260810 11:37:58.023165 14292 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:58 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361878, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:58.030807 14292 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260810 11:37:58.032423 14292 webserver.cc:460] Webserver started at http://127.3.240.194:45423/ using document root <none> and password file <none>
I20260810 11:37:58.033154 14292 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:58.033324 14292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:58.039408 14292 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:58.042593 14313 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:58.044373 14292 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:37:58.044579 14292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "9c8090f25aa7434e856fb7949607deb7"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "c1a3144d9c523b46bd581c663ac019cf"
server_key_iv: "6c707f2930c4c9d8e01da3cf709a8f01"
server_key_version: "encryptionkey@0"
I20260810 11:37:58.045699 14292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:58.068274 14292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:58.073230 14292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:58.073638 14292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:58.074651 14292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:58.076202 14292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:58.076327 14292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:58.076467 14292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:58.076556 14292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:58.114161 14292 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:43305
I20260810 11:37:58.114253 14480 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:43305 every 8 connection(s)
I20260810 11:37:58.115928 14292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:37:58.117149 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 14292
I20260810 11:37:58.117515 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 13329
Aug 10 11:37:58 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361872, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.195@KRBTEST.COM
Aug 10 11:37:58 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361878, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260810 11:37:58.152321 14488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.119014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47398 (local address 127.3.240.195:33373)
0810 11:37:58.123816 (+ 4802us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:58.123829 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:58.123867 (+ 38us) server_negotiation.cc:407] Connection header received
0810 11:37:58.123969 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:58.123980 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:58.124275 (+ 295us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:58.124508 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:58.127737 (+ 3229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.129489 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.131775 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.132743 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.133590 (+ 847us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.141035 (+ 7445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:58.141051 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:58.141067 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:58.141115 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:58.146128 (+ 5013us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.147743 (+ 1615us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.147756 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.147759 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.148016 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.149010 (+ 994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.149019 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.149023 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.149577 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:58.149936 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:58.151307 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0810 11:37:58.151737 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4578,"thread_start_us":4377,"threads_started":1}
I20260810 11:37:58.154222 13500 generic_service.cc:196] Checking for leaks (request via RPC)
Aug 10 11:37:58 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361878, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
I20260810 11:37:58.164124 14489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.120034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.3.240.194:34973)
0810 11:37:58.125745 (+ 5711us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:58.125807 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:58.131474 (+ 5667us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:58.133124 (+ 1650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:58.133137 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:58.134368 (+ 1231us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:58.135617 (+ 1249us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.135649 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.137556 (+ 1907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.137564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.140588 (+ 3024us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.140607 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.142277 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.142301 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.142892 (+ 591us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.143766 (+ 874us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:58.143805 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:58.149984 (+ 6179us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:58.155972 (+ 5988us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.155988 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.156012 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.157391 (+ 1379us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.159134 (+ 1743us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.159143 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.159150 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.159698 (+ 548us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.160417 (+ 719us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:58.160424 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:58.161090 (+ 666us) client_negotiation.cc:769] Sending connection context
0810 11:37:58.161463 (+ 373us) client_negotiation.cc:240] Negotiation successful
0810 11:37:58.162077 (+ 614us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5222,"thread_start_us":989,"threads_started":1}
I20260810 11:37:58.165498 14491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.121367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34973 (local address 127.3.240.253:34331)
0810 11:37:58.132278 (+ 10911us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:58.132285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:58.132306 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:37:58.132371 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:58.132376 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:58.132500 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:58.132621 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:58.135880 (+ 3259us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.137210 (+ 1330us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.140823 (+ 3613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.141650 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.141798 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.150396 (+ 8598us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:58.150408 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:58.150416 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:58.150481 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:58.155767 (+ 5286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.157746 (+ 1979us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.157755 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.157920 (+ 165us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.158846 (+ 926us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.159918 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.159925 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.159929 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.160263 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:58.160443 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:58.164757 (+ 4314us) server_negotiation.cc:299] Negotiation successful
0810 11:37:58.165063 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10773,"thread_start_us":10684,"threads_started":1}
I20260810 11:37:58.167169 14484 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34331
I20260810 11:37:58.167565 14484 heartbeater.cc:464] Registering TS with master...
Aug 10 11:37:58 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361878, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:37:58.168810 14484 heartbeater.cc:511] Master 127.3.240.253:34331 requested a full tablet report, sending...
I20260810 11:37:58.171249 13713 ts_manager.cc:194] Registered new tserver with Master: 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305)
I20260810 11:37:58.177650 14494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.128541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:48451)
0810 11:37:58.137154 (+ 8613us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:58.137204 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:58.137479 (+ 275us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:58.138669 (+ 1190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:58.138678 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:58.146829 (+ 8151us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:58.148177 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.148199 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.151098 (+ 2899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.151106 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.154699 (+ 3593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.154716 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.162335 (+ 7619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.162342 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.162563 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.163582 (+ 1019us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:58.163608 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:58.168988 (+ 5380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:58.174209 (+ 5221us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.174214 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.174218 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.174870 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.175428 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.175438 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.175446 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.175673 (+ 227us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.176416 (+ 743us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:58.176421 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:58.176607 (+ 186us) client_negotiation.cc:769] Sending connection context
0810 11:37:58.176774 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 11:37:58.177127 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8265,"thread_start_us":13391,"threads_started":1}
I20260810 11:37:58.178915 14493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.128761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48451 (local address 127.3.240.254:38543)
0810 11:37:58.129302 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:58.129307 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:58.137664 (+ 8357us) server_negotiation.cc:407] Connection header received
0810 11:37:58.137793 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:58.137802 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:58.137931 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:58.138085 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:58.148453 (+ 10368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.150821 (+ 2368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.155511 (+ 4690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.161877 (+ 6366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.162024 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.169294 (+ 7270us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:58.169302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:58.169307 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:58.169354 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:58.174033 (+ 4679us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.175071 (+ 1038us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.175077 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.175081 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.175282 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.175851 (+ 569us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.175858 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.175863 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.176264 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:58.177938 (+ 1674us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:58.178068 (+ 130us) server_negotiation.cc:299] Negotiation successful
0810 11:37:58.178304 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":417,"thread_start_us":290,"threads_started":1}
I20260810 11:37:58.179838 14481 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:58.180011 14481 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:58.180521 14481 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
I20260810 11:37:58.181661 12842 ts_manager.cc:194] Re-registered known tserver with Master: 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305)
I20260810 11:37:58.182884 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:48451
I20260810 11:37:58.186125 14492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.121039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.3.240.194:59031)
0810 11:37:58.132173 (+ 11134us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:58.132212 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:58.132495 (+ 283us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:58.133337 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:58.133346 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:58.135624 (+ 2278us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:58.136195 (+ 571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.136210 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.141981 (+ 5771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.141986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.144837 (+ 2851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:58.144861 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.146832 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.146842 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:58.147140 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.149080 (+ 1940us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:58.149108 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:58.172562 (+ 23454us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:58.177561 (+ 4999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.177572 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.177576 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.181838 (+ 4262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.182619 (+ 781us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:58.182624 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:58.182627 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:58.182796 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:58.185330 (+ 2534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:58.185334 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:58.185452 (+ 118us) client_negotiation.cc:769] Sending connection context
0810 11:37:58.185562 (+ 110us) client_negotiation.cc:240] Negotiation successful
0810 11:37:58.185806 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7338,"mutex_wait_us":4364,"thread_start_us":2563,"threads_started":1}
I20260810 11:37:58.186470 14490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:58.121006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:59031 (local address 127.3.240.252:39553)
0810 11:37:58.125312 (+ 4306us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:58.125317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:58.132392 (+ 7075us) server_negotiation.cc:407] Connection header received
0810 11:37:58.132693 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:58.132703 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:58.132858 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:58.133008 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:58.139613 (+ 6605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.141628 (+ 2015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.145145 (+ 3517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:58.146218 (+ 1073us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:58.146353 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:58.172879 (+ 26526us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:58.172887 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:58.172893 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:58.172941 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:58.177373 (+ 4432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.182267 (+ 4894us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.182273 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.182276 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.182463 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:58.182970 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:58.182975 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:58.182978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:58.183465 (+ 487us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:58.183662 (+ 197us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:58.185776 (+ 2114us) server_negotiation.cc:299] Negotiation successful
0810 11:37:58.186025 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4190,"thread_start_us":4036,"threads_started":1}
I20260810 11:37:58.187418 14483 heartbeater.cc:347] Connected to a master server at 127.3.240.252:39553
I20260810 11:37:58.187623 14483 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:58.188151 14483 heartbeater.cc:511] Master 127.3.240.252:39553 requested a full tablet report, sending...
I20260810 11:37:58.189924 13996 ts_manager.cc:194] Registered new tserver with Master: 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305)
I20260810 11:37:58.317233 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 13329
I20260810 11:37:58.363327 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:33373
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:38543
--webserver_port=33097
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34843
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:37:58.484663 14264 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
W20260810 11:37:59.025318 14502 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:37:59.025825 14502 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:37:59.025965 14502 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:37:59.038770 14502 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:37:59.039038 14502 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:37:59.039263 14502 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:37:59.059091 14502 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34843
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:33373
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=33097
--tserver_master_addrs=127.3.240.254:38543,127.3.240.253:34331,127.3.240.252:39553
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:37:59.063495 14502 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:37:59.065701 14502 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:37:59.089782 14510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:59.089891 14511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:37:59.089854 14513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:37:59.091167 14502 server_base.cc:1034] running on GCE node
I20260810 11:37:59.092370 14502 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:37:59.094436 14502 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:37:59.095866 14502 hybrid_clock.cc:648] HybridClock initialized: now 1786361879095701 us; error 111 us; skew 500 ppm
I20260810 11:37:59.096773 14502 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:37:59 dist-test-slave-drl0 krb5kdc[12776](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361879, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:37:59.102599 14502 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260810 11:37:59.105309 14502 webserver.cc:460] Webserver started at http://127.3.240.195:33097/ using document root <none> and password file <none>
I20260810 11:37:59.106572 14502 fs_manager.cc:359] Metadata directory not provided
I20260810 11:37:59.106878 14502 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:37:59.117796 14502 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.002s sys 0.006s
I20260810 11:37:59.124030 14524 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:59.126645 14502 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260810 11:37:59.126856 14502 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "d812df4cd66145dd8e713375f6ad2e3d"
format_stamp: "Formatted at 2026-08-10 11:37:54 on dist-test-slave-drl0"
server_key: "2e8b4e8295faa1e8c87dbfb34e73fa3f"
server_key_iv: "56ea9a3d34373403531242160c666f11"
server_key_version: "encryptionkey@0"
I20260810 11:37:59.128878 14502 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:37:59.181044 14502 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:37:59.185675 14481 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
I20260810 11:37:59.188952 14502 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:37:59.189833 14502 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:37:59.192857 14502 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:37:59.196466 14502 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:37:59.196672 14502 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:59.196831 14502 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:37:59.197011 14502 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:37:59.263404 14502 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33373
I20260810 11:37:59.263556 14690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33373 every 8 connection(s)
I20260810 11:37:59.265882 14502 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:37:59.266467 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 14502
Aug 10 11:37:59 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361879, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
Aug 10 11:37:59 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361879, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.252@KRBTEST.COM
Aug 10 11:37:59 dist-test-slave-drl0 krb5kdc[12776](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361879, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.253@KRBTEST.COM
I20260810 11:37:59.370909 14702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.275179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37751 (local address 127.3.240.254:38543)
0810 11:37:59.275921 (+ 742us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.275929 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.307524 (+ 31595us) server_negotiation.cc:407] Connection header received
0810 11:37:59.307777 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.307798 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.308144 (+ 346us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.308333 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.314565 (+ 6232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.316977 (+ 2412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.331840 (+ 14863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.333070 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.333353 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.346482 (+ 13129us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.346491 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.346497 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.346571 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.356567 (+ 9996us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.364834 (+ 8267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.364869 (+ 35us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.364876 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.365272 (+ 396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.366158 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.366162 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.366164 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.366891 (+ 727us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.369435 (+ 2544us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.369735 (+ 300us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.370182 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":549,"thread_start_us":892,"threads_started":1}
I20260810 11:37:59.374105 14708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.279309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35703 (local address 127.3.240.252:39553)
0810 11:37:59.290736 (+ 11427us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.290749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.308631 (+ 17882us) server_negotiation.cc:407] Connection header received
0810 11:37:59.311880 (+ 3249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.311888 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.312033 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.312233 (+ 200us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.322880 (+ 10647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.325143 (+ 2263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.334761 (+ 9618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.336525 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.336804 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.350201 (+ 13397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.350212 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.350229 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.350306 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.362910 (+ 12604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.364650 (+ 1740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.364668 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.364672 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.364984 (+ 312us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.366240 (+ 1256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.366250 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.366253 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.367014 (+ 761us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.367324 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.369580 (+ 2256us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.372641 (+ 3061us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":579,"server-negotiator.queue_time_us":11139,"thread_start_us":19356,"threads_started":1}
I20260810 11:37:59.374603 14707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.275881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:37751)
0810 11:37:59.306982 (+ 31101us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.307145 (+ 163us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.307527 (+ 382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.309385 (+ 1858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.309410 (+ 25us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.312233 (+ 2823us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:59.314238 (+ 2005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.314301 (+ 63us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.318171 (+ 3870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.318178 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.331536 (+ 13358us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.331566 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.335304 (+ 3738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.335314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.335988 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.338582 (+ 2594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:59.338634 (+ 52us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:59.346023 (+ 7389us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:59.358905 (+ 12882us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.358963 (+ 58us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.359012 (+ 49us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.360760 (+ 1748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.365489 (+ 4729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.365508 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.365511 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.365961 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.367240 (+ 1279us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:59.367272 (+ 32us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:59.368372 (+ 1100us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.369039 (+ 667us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.372803 (+ 3764us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30785,"mutex_wait_us":3051,"spinlock_wait_cycles":41728,"thread_start_us":681,"threads_started":1}
I20260810 11:37:59.374835 14700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.279103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.3.240.195:35703)
0810 11:37:59.308418 (+ 29315us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.308436 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.311626 (+ 3190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.319486 (+ 7860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.319494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.321357 (+ 1863us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:59.322605 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.322633 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.330823 (+ 8190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.330849 (+ 26us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.334209 (+ 3360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.334240 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.337174 (+ 2934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.337186 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.337860 (+ 674us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.339754 (+ 1894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:59.339805 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:59.349763 (+ 9958us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:59.363306 (+ 13543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.363313 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.363318 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.364418 (+ 1100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.365227 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.365234 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.365237 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.365868 (+ 631us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.367475 (+ 1607us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:59.367482 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:59.368745 (+ 1263us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.369104 (+ 359us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.371971 (+ 2867us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29179,"mutex_wait_us":3034,"thread_start_us":27268,"threads_started":1}
I20260810 11:37:59.375516 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.275782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47327 (local address 127.3.240.253:34331)
0810 11:37:59.276644 (+ 862us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.276670 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.277449 (+ 779us) server_negotiation.cc:407] Connection header received
0810 11:37:59.296329 (+ 18880us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.296343 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.296567 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.296891 (+ 324us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.317441 (+ 20550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.319575 (+ 2134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.328328 (+ 8753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.330208 (+ 1880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.330409 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.353811 (+ 23402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.353831 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.353856 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.353949 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.363651 (+ 9702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.365307 (+ 1656us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.365327 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.365334 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.365639 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.366632 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.366638 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.366641 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.367488 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.367741 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.370034 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.370417 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":676,"thread_start_us":448,"threads_started":1}
I20260810 11:37:59.376540 14706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.273989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.3.240.195:47327)
0810 11:37:59.277171 (+ 3182us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.277248 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.292075 (+ 14827us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.299897 (+ 7822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.299942 (+ 45us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.302276 (+ 2334us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:37:59.315219 (+ 12943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.315247 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.320466 (+ 5219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.320474 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.328125 (+ 7651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.328156 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.337722 (+ 9566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.337746 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.338251 (+ 505us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.346194 (+ 7943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:37:59.346284 (+ 90us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:37:59.353293 (+ 7009us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:37:59.364027 (+ 10734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.364033 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.364037 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.365024 (+ 987us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.365929 (+ 905us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:37:59.365935 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:37:59.365949 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:37:59.366374 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:37:59.367824 (+ 1450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:37:59.367828 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:37:59.368372 (+ 544us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.369039 (+ 667us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.371728 (+ 2689us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2841,"mutex_wait_us":8,"spinlock_wait_cycles":63360,"thread_start_us":31656,"threads_started":1}
I20260810 11:37:59.379142 14692 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38543
I20260810 11:37:59.379249 14695 heartbeater.cc:347] Connected to a master server at 127.3.240.253:34331
I20260810 11:37:59.379797 14692 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:59.379798 14695 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:59.381991 14694 heartbeater.cc:347] Connected to a master server at 127.3.240.252:39553
I20260810 11:37:59.382298 14694 heartbeater.cc:464] Registering TS with master...
I20260810 11:37:59.382627 14692 heartbeater.cc:511] Master 127.3.240.254:38543 requested a full tablet report, sending...
I20260810 11:37:59.382817 14695 heartbeater.cc:511] Master 127.3.240.253:34331 requested a full tablet report, sending...
I20260810 11:37:59.383880 14694 heartbeater.cc:511] Master 127.3.240.252:39553 requested a full tablet report, sending...
I20260810 11:37:59.386839 13996 ts_manager.cc:194] Registered new tserver with Master: d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373)
I20260810 11:37:59.387070 12842 ts_manager.cc:194] Re-registered known tserver with Master: d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373)
I20260810 11:37:59.387735 13713 ts_manager.cc:194] Registered new tserver with Master: d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373)
I20260810 11:37:59.389217 12842 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:37751
I20260810 11:37:59.396981 14716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.320602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37440 (local address 127.3.240.252:39553)
0810 11:37:59.336172 (+ 15570us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.336190 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.336231 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:37:59.336400 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.336407 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.336629 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.336808 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.342743 (+ 5935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.349679 (+ 6936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.361357 (+ 11678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.363204 (+ 1847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.363444 (+ 240us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.374793 (+ 11349us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.374818 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.374830 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.374993 (+ 163us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.389641 (+ 14648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.391519 (+ 1878us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.391536 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.391542 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.391971 (+ 429us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.393097 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.393110 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.393113 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.393799 (+ 686us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.394377 (+ 578us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.395396 (+ 1019us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.395932 (+ 536us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":15252,"thread_start_us":14235,"threads_started":1}
I20260810 11:37:59.403520 14712 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.289883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51102 (local address 127.3.240.254:38543)
0810 11:37:59.319661 (+ 29778us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.319669 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.319712 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:37:59.319804 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.319816 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.319928 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.320052 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.330365 (+ 10313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.332529 (+ 2164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.341924 (+ 9395us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.343509 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.343758 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.353909 (+ 10151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.353933 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.353939 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.354108 (+ 169us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.395247 (+ 41139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.396718 (+ 1471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.396724 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.396727 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.396935 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.399073 (+ 2138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.399091 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.399095 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.399829 (+ 734us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.400197 (+ 368us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.401887 (+ 1690us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.402426 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29551,"thread_start_us":34664,"threads_started":1}
I20260810 11:37:59.407553 14714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.307190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35396 (local address 127.3.240.253:34331)
0810 11:37:59.329667 (+ 22477us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.329684 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.329729 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:37:59.329832 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.329851 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.329993 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.335195 (+ 5202us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:37:59.344753 (+ 9558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.347484 (+ 2731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.370084 (+ 22600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.376251 (+ 6167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.376629 (+ 378us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.391497 (+ 14868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:37:59.391514 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:37:59.391520 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:37:59.391577 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:37:59.399494 (+ 7917us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.400988 (+ 1494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.401008 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.401014 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.401373 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:37:59.403862 (+ 2489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:37:59.403879 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:37:59.403884 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:37:59.404590 (+ 706us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:37:59.405327 (+ 737us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.405891 (+ 564us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.406536 (+ 645us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22261,"thread_start_us":21958,"threads_started":1}
I20260810 11:37:59.411371 12842 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:37:59.431510 12842 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:37:59.526636 14737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.481876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:43305 (local address 127.0.0.1:39130)
0810 11:37:59.490920 (+ 9044us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.491075 (+ 155us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.491548 (+ 473us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.501622 (+ 10074us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.501630 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.501727 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:59.504377 (+ 2650us) spinlock_profiling.cc:243] Waited 1.3 ms on lock 0x61200002c628. stack: 00007f3a956f766e 00007f3a956f7430 00007f3a9477e5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f3a956ed0fd 00007f3a956ecd1e 00007f3aa85f63cd 00007f3aa85f636c 00007f3a96826757 00007f3a9c0fb03a 00007f3a9c182dca 00007f3a9c180f68 00007f3a9c1b482f 00007f3a9c1b458d
0810 11:37:59.505715 (+ 1338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.505747 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.510908 (+ 5161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.510945 (+ 37us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.518901 (+ 7956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.519066 (+ 165us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.524583 (+ 5517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.524602 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.524982 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.525052 (+ 70us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.525224 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.525623 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8720,"spinlock_wait_cycles":3003904,"thread_start_us":7462,"threads_started":1}
I20260810 11:37:59.527524 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.482485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39130 (local address 127.3.240.194:43305)
0810 11:37:59.500445 (+ 17960us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.500459 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.500507 (+ 48us) server_negotiation.cc:407] Connection header received
0810 11:37:59.500736 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.500755 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.501214 (+ 459us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.501683 (+ 469us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:59.507269 (+ 5586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.510577 (+ 3308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.519530 (+ 8953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.523734 (+ 4204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.524155 (+ 421us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.524873 (+ 718us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.525749 (+ 876us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.526797 (+ 1048us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17608,"thread_start_us":17363,"threads_started":1}
I20260810 11:37:59.531389 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.498499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60388 (local address 127.3.240.193:38681)
0810 11:37:59.510555 (+ 12056us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.510566 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.511299 (+ 733us) server_negotiation.cc:407] Connection header received
0810 11:37:59.511691 (+ 392us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.511708 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.511999 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.512473 (+ 474us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:59.517038 (+ 4565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.518831 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.525078 (+ 6247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.528530 (+ 3452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.528735 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.528978 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.530335 (+ 1357us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.530702 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11753,"thread_start_us":11505,"threads_started":1}
I20260810 11:37:59.531687 14739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.482592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33373 (local address 127.0.0.1:47414)
0810 11:37:59.498800 (+ 16208us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.498848 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.499180 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.501595 (+ 2415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.501603 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.501666 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:59.507081 (+ 5415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.507102 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.512476 (+ 5374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.512483 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.524202 (+ 11719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.524220 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.529795 (+ 5575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.529818 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.530248 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.530358 (+ 110us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.530526 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.530786 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15814,"mutex_wait_us":144,"thread_start_us":18599,"threads_started":1}
I20260810 11:37:59.533102 14740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.483287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47414 (local address 127.3.240.195:33373)
0810 11:37:59.496910 (+ 13623us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.496941 (+ 31us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.500155 (+ 3214us) server_negotiation.cc:407] Connection header received
0810 11:37:59.500403 (+ 248us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.500428 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.501064 (+ 636us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.502829 (+ 1765us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:59.507785 (+ 4956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.512146 (+ 4361us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.524566 (+ 12420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.529252 (+ 4686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.529515 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.530015 (+ 500us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.532033 (+ 2018us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.532364 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12975,"thread_start_us":12455,"threads_started":1}
I20260810 11:37:59.534053 14741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.498234 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38681 (local address 127.0.0.1:60388)
0810 11:37:59.511086 (+ 12852us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.511119 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.511404 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.512453 (+ 1049us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.512471 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.512500 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:59.516247 (+ 3747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.516280 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.519111 (+ 2831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.519120 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.524544 (+ 5424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.524597 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.529166 (+ 4569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.529191 (+ 25us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.529798 (+ 607us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.529897 (+ 99us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.530168 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.532449 (+ 2281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12584,"mutex_wait_us":12,"thread_start_us":11073,"threads_started":1}
I20260810 11:37:59.536613 14386 tablet_service.cc:1467] Processing CreateTablet for tablet eee027c3b3334957999ebb72b3300b98 (DEFAULT_TABLE table=test-table [id=db49801f7c5e41ea8af33a57e2d4529b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:59.539435 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee027c3b3334957999ebb72b3300b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:59.540612 14596 tablet_service.cc:1467] Processing CreateTablet for tablet eee027c3b3334957999ebb72b3300b98 (DEFAULT_TABLE table=test-table [id=db49801f7c5e41ea8af33a57e2d4529b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:59.541196 14173 tablet_service.cc:1467] Processing CreateTablet for tablet eee027c3b3334957999ebb72b3300b98 (DEFAULT_TABLE table=test-table [id=db49801f7c5e41ea8af33a57e2d4529b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:37:59.543823 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee027c3b3334957999ebb72b3300b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:59.543871 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eee027c3b3334957999ebb72b3300b98. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:37:59.573154 14744 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Bootstrap starting.
I20260810 11:37:59.581604 14744 tablet_bootstrap.cc:654] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:59.584760 14748 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Bootstrap starting.
I20260810 11:37:59.586294 14744 log.cc:824] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:59.589390 14752 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Bootstrap starting.
I20260810 11:37:59.597018 14752 tablet_bootstrap.cc:654] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:59.597752 14748 tablet_bootstrap.cc:654] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Neither blocks nor log segments found. Creating new log.
I20260810 11:37:59.598839 14744 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: No bootstrap required, opened a new log
I20260810 11:37:59.599598 14752 log.cc:824] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:59.599931 14744 ts_tablet_manager.cc:1397] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Time spent bootstrapping tablet: real 0.027s user 0.023s sys 0.000s
I20260810 11:37:59.602221 14748 log.cc:824] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Log is configured to *not* fsync() on all Append() calls
I20260810 11:37:59.606002 14752 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: No bootstrap required, opened a new log
I20260810 11:37:59.606304 14752 ts_tablet_manager.cc:1397] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Time spent bootstrapping tablet: real 0.018s user 0.013s sys 0.000s
I20260810 11:37:59.607944 14748 tablet_bootstrap.cc:492] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: No bootstrap required, opened a new log
I20260810 11:37:59.610672 14748 ts_tablet_manager.cc:1397] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Time spent bootstrapping tablet: real 0.026s user 0.007s sys 0.013s
I20260810 11:37:59.621415 14744 raft_consensus.cc:348] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.622721 14744 raft_consensus.cc:374] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:59.623078 14744 raft_consensus.cc:729] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8090f25aa7434e856fb7949607deb7, State: Initialized, Role: FOLLOWER
I20260810 11:37:59.624338 14744 consensus_queue.cc:260] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.628613 14744 ts_tablet_manager.cc:1428] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Time spent starting tablet: real 0.028s user 0.020s sys 0.008s
I20260810 11:37:59.633493 14752 raft_consensus.cc:348] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.634735 14752 raft_consensus.cc:374] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:59.635161 14752 raft_consensus.cc:729] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d812df4cd66145dd8e713375f6ad2e3d, State: Initialized, Role: FOLLOWER
I20260810 11:37:59.636747 14752 consensus_queue.cc:260] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.641634 14692 heartbeater.cc:503] Master 127.3.240.254:38543 was elected leader, sending a full tablet report...
I20260810 11:37:59.641448 14748 raft_consensus.cc:348] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.642614 14748 raft_consensus.cc:374] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:37:59.642829 14748 raft_consensus.cc:729] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6160c92c581a455798298753a797bbfb, State: Initialized, Role: FOLLOWER
I20260810 11:37:59.643389 14752 ts_tablet_manager.cc:1428] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Time spent starting tablet: real 0.037s user 0.020s sys 0.018s
I20260810 11:37:59.644316 14748 consensus_queue.cc:260] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.656543 14748 ts_tablet_manager.cc:1428] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Time spent starting tablet: real 0.045s user 0.028s sys 0.014s
W20260810 11:37:59.680846 14270 tablet.cc:2369] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:59.775004 14697 tablet.cc:2369] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:37:59.872243 14486 tablet.cc:2369] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:37:59.949035 14759 raft_consensus.cc:482] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:37:59.949782 14759 raft_consensus.cc:504] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.952916 14759 leader_election.cc:288] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305), 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:59.971865 14738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.953446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57839 (local address 127.3.240.194:43305)
0810 11:37:59.954000 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.954010 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.955717 (+ 1707us) server_negotiation.cc:407] Connection header received
0810 11:37:59.956339 (+ 622us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.956351 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.956513 (+ 162us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.956667 (+ 154us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:59.957787 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.961015 (+ 3228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.966158 (+ 5143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.969307 (+ 3149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.969700 (+ 393us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.969840 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.970743 (+ 903us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.971347 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 11:37:59.972514 14762 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.953369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:43305 (local address 127.3.240.195:57839)
0810 11:37:59.955526 (+ 2157us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.955556 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.956001 (+ 445us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.956711 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.956716 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.956771 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:59.957336 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.957367 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.961409 (+ 4042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.961429 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.965817 (+ 4388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.965830 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.969738 (+ 3908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.969747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.970276 (+ 529us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.970440 (+ 164us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.970641 (+ 201us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.971225 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1640,"mutex_wait_us":1,"thread_start_us":1136,"threads_started":1}
I20260810 11:37:59.975054 14743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.954663 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41077 (local address 127.3.240.193:38681)
0810 11:37:59.955312 (+ 649us) server_negotiation.cc:206] Beginning negotiation
0810 11:37:59.955320 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:37:59.955787 (+ 467us) server_negotiation.cc:407] Connection header received
0810 11:37:59.956512 (+ 725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:37:59.956522 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:37:59.956983 (+ 461us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:37:59.957228 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:37:59.958268 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.959604 (+ 1336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.964987 (+ 5383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.968314 (+ 3327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.969752 (+ 1438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.970014 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 11:37:59.972412 (+ 2398us) server_negotiation.cc:299] Negotiation successful
0810 11:37:59.972722 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145}
I20260810 11:37:59.973263 14763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:37:59.953977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38681 (local address 127.3.240.195:41077)
0810 11:37:59.955655 (+ 1678us) negotiation.cc:107] Waiting for socket to connect
0810 11:37:59.955679 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:37:59.956019 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:37:59.957299 (+ 1280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:37:59.957306 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:37:59.957357 (+ 51us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:37:59.957987 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.958013 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.959986 (+ 1973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.960005 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.964315 (+ 4310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:37:59.964443 (+ 128us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:37:59.970119 (+ 5676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:37:59.970136 (+ 17us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:37:59.970551 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:37:59.970641 (+ 90us) client_negotiation.cc:769] Sending connection context
0810 11:37:59.971135 (+ 494us) client_negotiation.cc:240] Negotiation successful
0810 11:37:59.971305 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1488,"thread_start_us":1474,"threads_started":1}
I20260810 11:37:59.976599 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee027c3b3334957999ebb72b3300b98" candidate_uuid: "d812df4cd66145dd8e713375f6ad2e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8090f25aa7434e856fb7949607deb7" is_pre_election: true
I20260810 11:37:59.978710 14417 raft_consensus.cc:2454] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d812df4cd66145dd8e713375f6ad2e3d in term 0.
I20260810 11:37:59.978639 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee027c3b3334957999ebb72b3300b98" candidate_uuid: "d812df4cd66145dd8e713375f6ad2e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6160c92c581a455798298753a797bbfb" is_pre_election: true
I20260810 11:37:59.979929 14203 raft_consensus.cc:2454] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d812df4cd66145dd8e713375f6ad2e3d in term 0.
I20260810 11:37:59.980423 14527 leader_election.cc:302] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9c8090f25aa7434e856fb7949607deb7, d812df4cd66145dd8e713375f6ad2e3d; no voters:
I20260810 11:37:59.981973 14759 raft_consensus.cc:2781] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:37:59.982311 14759 raft_consensus.cc:482] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:37:59.982551 14759 raft_consensus.cc:3037] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:59.987370 14759 raft_consensus.cc:504] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:37:59.988529 14759 leader_election.cc:288] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [CANDIDATE]: Term 1 election: Requested vote from peers 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305), 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:37:59.990119 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee027c3b3334957999ebb72b3300b98" candidate_uuid: "d812df4cd66145dd8e713375f6ad2e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6160c92c581a455798298753a797bbfb"
I20260810 11:37:59.989945 14417 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eee027c3b3334957999ebb72b3300b98" candidate_uuid: "d812df4cd66145dd8e713375f6ad2e3d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9c8090f25aa7434e856fb7949607deb7"
I20260810 11:37:59.990881 14203 raft_consensus.cc:3037] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:59.991600 14417 raft_consensus.cc:3037] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:37:59.997330 14203 raft_consensus.cc:2454] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d812df4cd66145dd8e713375f6ad2e3d in term 1.
I20260810 11:37:59.998260 14526 leader_election.cc:302] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6160c92c581a455798298753a797bbfb, d812df4cd66145dd8e713375f6ad2e3d; no voters:
I20260810 11:37:59.999123 14759 raft_consensus.cc:2781] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:00.000046 14417 raft_consensus.cc:2454] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d812df4cd66145dd8e713375f6ad2e3d in term 1.
I20260810 11:38:00.000233 14759 raft_consensus.cc:686] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 1 LEADER]: Becoming Leader. State: Replica: d812df4cd66145dd8e713375f6ad2e3d, State: Running, Role: LEADER
I20260810 11:38:00.001257 14759 consensus_queue.cc:237] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:00.013629 12842 catalog_manager.cc:5495] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d reported cstate change: term changed from 0 to 1, leader changed from <none> to d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195). New cstate: current_term: 1 leader_uuid: "d812df4cd66145dd8e713375f6ad2e3d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:00.121098 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:00.107479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47416 (local address 127.3.240.195:33373)
0810 11:38:00.109110 (+ 1631us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:00.109125 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:00.109174 (+ 49us) server_negotiation.cc:407] Connection header received
0810 11:38:00.109316 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:00.109321 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:00.109485 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:00.109927 (+ 442us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:00.110497 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.112526 (+ 2029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.115080 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.116612 (+ 1532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.116872 (+ 260us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:00.117755 (+ 883us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:00.119525 (+ 1770us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:00.119899 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:00.120177 (+ 278us) server_negotiation.cc:299] Negotiation successful
0810 11:38:00.120354 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1379,"thread_start_us":912,"threads_started":1}
I20260810 11:38:00.136909 14203 raft_consensus.cc:1261] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Refusing update from remote peer d812df4cd66145dd8e713375f6ad2e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:00.136986 14417 raft_consensus.cc:1261] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Refusing update from remote peer d812df4cd66145dd8e713375f6ad2e3d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:00.139227 14759 consensus_queue.cc:1035] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:00.139806 14764 consensus_queue.cc:1035] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [LEADER]: Connected to new peer: Peer: permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:00.219758 12841 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:00.220337 12841 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51102:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:00.232708 12841 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 57926610f9004c339176098dcb8abf42: Sending DeleteTablet for 3 replicas of tablet eee027c3b3334957999ebb72b3300b98
I20260810 11:38:00.234886 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet eee027c3b3334957999ebb72b3300b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:00 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39130
I20260810 11:38:00.235483 14173 tablet_service.cc:1514] Processing DeleteTablet for tablet eee027c3b3334957999ebb72b3300b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:00 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:60388
I20260810 11:38:00.236951 14596 tablet_service.cc:1514] Processing DeleteTablet for tablet eee027c3b3334957999ebb72b3300b98 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:00 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:47414
I20260810 11:38:00.239802 14796 tablet_replica.cc:330] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: stopping tablet replica
I20260810 11:38:00.240772 14796 raft_consensus.cc:2229] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:00.242015 14796 raft_consensus.cc:2258] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:00.245471 14799 tablet_replica.cc:330] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: stopping tablet replica
I20260810 11:38:00.246630 14799 raft_consensus.cc:2229] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:00.248450 14798 tablet_replica.cc:330] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: stopping tablet replica
I20260810 11:38:00.249020 14798 raft_consensus.cc:2229] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:00.249810 14798 raft_consensus.cc:2258] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:00.251691 14799 raft_consensus.cc:2258] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:00.256436 14799 ts_tablet_manager.cc:1905] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:00.257151 14798 ts_tablet_manager.cc:1905] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:00.264413 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 12795
I20260810 11:38:00.264497 14796 ts_tablet_manager.cc:1905] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:00.267944 12886 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:00.285292 14796 ts_tablet_manager.cc:1918] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:00.285666 14796 log.cc:1196] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/eee027c3b3334957999ebb72b3300b98
I20260810 11:38:00.286921 14796 ts_tablet_manager.cc:1939] T eee027c3b3334957999ebb72b3300b98 P 9c8090f25aa7434e856fb7949607deb7: Deleting consensus metadata
I20260810 11:38:00.298357 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:00.239959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35410 (local address 127.3.240.253:34331)
0810 11:38:00.244535 (+ 4576us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:00.244555 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:00.254437 (+ 9882us) server_negotiation.cc:407] Connection header received
0810 11:38:00.254952 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:00.254974 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:00.255128 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:00.255299 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:00.260718 (+ 5419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.263005 (+ 2287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.268944 (+ 5939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.270162 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.270383 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:00.280264 (+ 9881us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:00.280284 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:00.280291 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:00.280386 (+ 95us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:00.289777 (+ 9391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.291971 (+ 2194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.291979 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.291983 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.292511 (+ 528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.293677 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.293683 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.293686 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.294534 (+ 848us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:00.294881 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:00.297177 (+ 2296us) server_negotiation.cc:299] Negotiation successful
0810 11:38:00.297635 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4281,"thread_start_us":4007,"threads_started":1}
I20260810 11:38:00.304482 14798 ts_tablet_manager.cc:1918] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:00.304955 14798 log.cc:1196] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/eee027c3b3334957999ebb72b3300b98
I20260810 11:38:00.306002 14798 ts_tablet_manager.cc:1939] T eee027c3b3334957999ebb72b3300b98 P 6160c92c581a455798298753a797bbfb: Deleting consensus metadata
I20260810 11:38:00.310662 14799 ts_tablet_manager.cc:1918] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:00.311136 14799 log.cc:1196] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/eee027c3b3334957999ebb72b3300b98
I20260810 11:38:00.313457 14799 ts_tablet_manager.cc:1939] T eee027c3b3334957999ebb72b3300b98 P d812df4cd66145dd8e713375f6ad2e3d: Deleting consensus metadata
I20260810 11:38:00.336056 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:00.254350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37448 (local address 127.3.240.252:39553)
0810 11:38:00.269249 (+ 14899us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:00.269271 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:00.301347 (+ 32076us) server_negotiation.cc:407] Connection header received
0810 11:38:00.301974 (+ 627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:00.301995 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:00.302252 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:00.302454 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:00.307292 (+ 4838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.308552 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.313245 (+ 4693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.314691 (+ 1446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.315002 (+ 311us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:00.321948 (+ 6946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:00.321969 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:00.321975 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:00.322061 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:00.328744 (+ 6683us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.330274 (+ 1530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.330280 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.330284 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.331140 (+ 856us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.331922 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.331928 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.331932 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.332571 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:00.334618 (+ 2047us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:00.334912 (+ 294us) server_negotiation.cc:299] Negotiation successful
0810 11:38:00.335353 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14562,"thread_start_us":14933,"threads_started":1}
I20260810 11:38:00.537282 12821 catalog_manager.cc:4841] TS 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305): tablet eee027c3b3334957999ebb72b3300b98 (table test-table [id=db49801f7c5e41ea8af33a57e2d4529b]) successfully deleted
I20260810 11:38:00.537613 12820 catalog_manager.cc:4841] TS 6160c92c581a455798298753a797bbfb (127.3.240.193:38681): tablet eee027c3b3334957999ebb72b3300b98 (table test-table [id=db49801f7c5e41ea8af33a57e2d4529b]) successfully deleted
I20260810 11:38:00.537961 12824 catalog_manager.cc:4841] TS d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373): tablet eee027c3b3334957999ebb72b3300b98 (table test-table [id=db49801f7c5e41ea8af33a57e2d4529b]) successfully deleted
I20260810 11:38:00.539018 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 12795
I20260810 11:38:00.654837 14805 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:00.623897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37456 (local address 127.3.240.252:39553)
0810 11:38:00.624308 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:00.624318 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:00.625031 (+ 713us) server_negotiation.cc:407] Connection header received
0810 11:38:00.625426 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:00.625446 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:00.625587 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:00.625762 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:00.630030 (+ 4268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.631271 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.634342 (+ 3071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.635790 (+ 1448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.636039 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:00.643163 (+ 7124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:00.643169 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:00.643173 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:00.643230 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:00.647952 (+ 4722us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.649123 (+ 1171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.649142 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.649146 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.649456 (+ 310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.650353 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.650360 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.650365 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.651158 (+ 793us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:00.651542 (+ 384us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:00.653288 (+ 1746us) server_negotiation.cc:299] Negotiation successful
0810 11:38:00.653839 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260810 11:38:00.658334 14804 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:00.622261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35414 (local address 127.3.240.253:34331)
0810 11:38:00.622642 (+ 381us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:00.622661 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:00.624125 (+ 1464us) server_negotiation.cc:407] Connection header received
0810 11:38:00.625189 (+ 1064us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:00.625196 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:00.625341 (+ 145us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:00.625503 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:00.628841 (+ 3338us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.630057 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.633467 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:00.635264 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:00.635593 (+ 329us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:00.643023 (+ 7430us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:00.643034 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:00.643041 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:00.643116 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:00.649434 (+ 6318us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.650867 (+ 1433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.650873 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.650875 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.651211 (+ 336us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:00.652874 (+ 1663us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:00.652886 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:00.652890 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:00.653781 (+ 891us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:00.654454 (+ 673us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:00.656923 (+ 2469us) server_negotiation.cc:299] Negotiation successful
0810 11:38:00.657533 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260810 11:38:01.202327 14814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.200186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:41071)
0810 11:38:01.201409 (+ 1223us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.201776 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":959,"thread_start_us":542,"threads_started":1}
W20260810 11:38:01.203260 14692 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.215901 14815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.211611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:42765)
0810 11:38:01.214599 (+ 2988us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.215497 (+ 898us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2784,"thread_start_us":2530,"threads_started":1}
W20260810 11:38:01.217476 14264 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.218843 14816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.217660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:46265)
0810 11:38:01.218423 (+ 763us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.218600 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":602,"thread_start_us":411,"threads_started":1}
W20260810 11:38:01.219651 14481 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:38543 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.731114 14819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57926610f9004c339176098dcb8abf42)
I20260810 11:38:01.731621 14819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } }
I20260810 11:38:01.731696 14822 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 57926610f9004c339176098dcb8abf42)
I20260810 11:38:01.732317 14822 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } }
I20260810 11:38:01.736922 14819 leader_election.cc:288] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543), ee426fdc32f54c42b732c4d50632a67e (127.3.240.253:34331)
I20260810 11:38:01.739684 14822 leader_election.cc:288] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543), 970e8a6cd0a840ab99fabcf63b58ee1f (127.3.240.252:39553)
I20260810 11:38:01.740391 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.735138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51540)
0810 11:38:01.739433 (+ 4295us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.739764 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4090,"thread_start_us":2362,"threads_started":1}
W20260810 11:38:01.741027 13981 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.740638 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.734275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51532)
0810 11:38:01.739638 (+ 5363us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.739964 (+ 326us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5137,"thread_start_us":4885,"threads_started":1}
W20260810 11:38:01.743561 13698 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.753376 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.752439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51564)
0810 11:38:01.752723 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.753011 (+ 288us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 11:38:01.753798 14833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.751185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51556)
0810 11:38:01.752896 (+ 1711us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.753313 (+ 417us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1474,"threads_started":1}
W20260810 11:38:01.753919 13981 leader_election.cc:334] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543): Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
W20260810 11:38:01.754383 13698 leader_election.cc:334] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543): Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.756928 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.735083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34331 (local address 127.0.0.1:35424)
0810 11:38:01.738129 (+ 3046us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.738184 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.738450 (+ 266us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.741158 (+ 2708us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.741163 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.741191 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.742217 (+ 1026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.742264 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.747068 (+ 4804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.747091 (+ 23us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.751597 (+ 4506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.751652 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.755439 (+ 3787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.755449 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.755809 (+ 360us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.755851 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 11:38:01.756045 (+ 194us) client_negotiation.cc:240] Negotiation successful
0810 11:38:01.756304 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2700,"thread_start_us":3815,"threads_started":1}
I20260810 11:38:01.756984 14826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.739639 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35424 (local address 127.3.240.253:34331)
0810 11:38:01.740722 (+ 1083us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.740738 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.740781 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:38:01.740920 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.740924 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.741009 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.741131 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.742723 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.745133 (+ 2410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.752031 (+ 6898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.755136 (+ 3105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.755494 (+ 358us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.755716 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:01.756079 (+ 363us) server_negotiation.cc:299] Negotiation successful
0810 11:38:01.756304 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":31,"server-negotiator.queue_time_us":797,"thread_start_us":491,"threads_started":1}
I20260810 11:38:01.757882 13727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee426fdc32f54c42b732c4d50632a67e" is_pre_election: true
I20260810 11:38:01.758355 13727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 970e8a6cd0a840ab99fabcf63b58ee1f in term 1.
I20260810 11:38:01.759141 13978 leader_election.cc:302] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 970e8a6cd0a840ab99fabcf63b58ee1f, ee426fdc32f54c42b732c4d50632a67e; no voters: 57926610f9004c339176098dcb8abf42
I20260810 11:38:01.759759 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260810 11:38:01.760002 14819 raft_consensus.cc:482] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Starting leader election (detected failure of leader 57926610f9004c339176098dcb8abf42)
I20260810 11:38:01.760191 14819 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:38:01.761883 14823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.743054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39553 (local address 127.0.0.1:37470)
0810 11:38:01.745866 (+ 2812us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.745937 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.746434 (+ 497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.751146 (+ 4712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.751152 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.751191 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.752238 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.752268 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.755223 (+ 2955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.755232 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.757752 (+ 2520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.757791 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.760459 (+ 2668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.760480 (+ 21us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.760758 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.760806 (+ 48us) client_negotiation.cc:769] Sending connection context
0810 11:38:01.761027 (+ 221us) client_negotiation.cc:240] Negotiation successful
0810 11:38:01.761304 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2306,"mutex_wait_us":6}
I20260810 11:38:01.762148 14829 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.741044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37470 (local address 127.3.240.252:39553)
0810 11:38:01.749207 (+ 8163us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.749225 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.749260 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:38:01.749407 (+ 147us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.749462 (+ 55us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.749608 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.749827 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.752748 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.754988 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.758268 (+ 3280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.760130 (+ 1862us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.760383 (+ 253us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.760592 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:01.761086 (+ 494us) server_negotiation.cc:299] Negotiation successful
0810 11:38:01.761346 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":7893,"thread_start_us":7620,"threads_started":1}
I20260810 11:38:01.763157 14012 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee426fdc32f54c42b732c4d50632a67e" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" is_pre_election: true
I20260810 11:38:01.764567 14819 raft_consensus.cc:504] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } }
I20260810 11:38:01.765946 14012 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ee426fdc32f54c42b732c4d50632a67e in current term 2: Already voted for candidate 970e8a6cd0a840ab99fabcf63b58ee1f in this term.
I20260810 11:38:01.766527 14819 leader_election.cc:288] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 election: Requested vote from peers 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543), ee426fdc32f54c42b732c4d50632a67e (127.3.240.253:34331)
I20260810 11:38:01.767771 13727 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "ee426fdc32f54c42b732c4d50632a67e"
I20260810 11:38:01.769239 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.766254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51566)
0810 11:38:01.767581 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.768092 (+ 511us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":168}
I20260810 11:38:01.769150 13727 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:38:01.770475 13698 leader_election.cc:302] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ee426fdc32f54c42b732c4d50632a67e; no voters: 57926610f9004c339176098dcb8abf42, 970e8a6cd0a840ab99fabcf63b58ee1f
I20260810 11:38:01.772037 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.771333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51580)
0810 11:38:01.771586 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.771782 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260810 11:38:01.772440 13981 leader_election.cc:334] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543): Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
I20260810 11:38:01.774159 13727 raft_consensus.cc:2454] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 970e8a6cd0a840ab99fabcf63b58ee1f in term 2.
I20260810 11:38:01.774390 14822 raft_consensus.cc:2726] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:38:01.775043 13978 leader_election.cc:302] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 970e8a6cd0a840ab99fabcf63b58ee1f, ee426fdc32f54c42b732c4d50632a67e; no voters: 57926610f9004c339176098dcb8abf42
I20260810 11:38:01.775712 14819 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 2 FOLLOWER]: Leader election won for term 2
I20260810 11:38:01.776525 14819 raft_consensus.cc:686] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [term 2 LEADER]: Becoming Leader. State: Replica: 970e8a6cd0a840ab99fabcf63b58ee1f, State: Running, Role: LEADER
I20260810 11:38:01.777145 14819 consensus_queue.cc:237] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } }
I20260810 11:38:01.781189 14834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970e8a6cd0a840ab99fabcf63b58ee1f. Latest consensus state: current_term: 2 leader_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:38:01.781957 14834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:01.783365 14836 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:01.793458 14836 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:01.795104 14836 catalog_manager.cc:1259] Loaded cluster ID: dee69dbb8d9348d097481bbd6c939eb8
I20260810 11:38:01.795231 14836 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:01.797361 14836 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:01.798857 14836 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Loaded TSK: 0
I20260810 11:38:01.800401 14836 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:38:01.853346 13996 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37456:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:01.865599 13996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:01.873699 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.872874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51604)
0810 11:38:01.873129 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.873304 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 11:38:01.873782 13727 raft_consensus.cc:1261] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [term 2 FOLLOWER]: Refusing update from remote peer 970e8a6cd0a840ab99fabcf63b58ee1f: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260810 11:38:01.875046 14834 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
I20260810 11:38:01.875638 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.874902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51618)
0810 11:38:01.875186 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.875332 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260810 11:38:01.875984 13981 consensus_peers.cc:487] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f -> Peer 57926610f9004c339176098dcb8abf42 (127.3.240.254:38543): Couldn't send request to peer 57926610f9004c339176098dcb8abf42. Status: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 11:38:01.879513 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 970e8a6cd0a840ab99fabcf63b58ee1f. Latest consensus state: current_term: 2 leader_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:38:01.880084 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:38:01.883353 14834 sys_catalog.cc:455] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:38:01.883844 14834 sys_catalog.cc:458] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:01.886301 14822 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "57926610f9004c339176098dcb8abf42" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38543 } } peers { permanent_uuid: "ee426fdc32f54c42b732c4d50632a67e" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34331 } attrs { promote: false } } peers { permanent_uuid: "970e8a6cd0a840ab99fabcf63b58ee1f" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39553 } attrs { promote: false } } }
I20260810 11:38:01.886865 14822 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee426fdc32f54c42b732c4d50632a67e [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:38:01.913039 14825 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.900039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33373 (local address 127.0.0.1:47418)
0810 11:38:01.900269 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.900305 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.900500 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.901778 (+ 1278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.901800 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.901858 (+ 58us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.902299 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.902329 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.904597 (+ 2268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.904615 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.908057 (+ 3442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.908071 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.911561 (+ 3490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.911569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.911914 (+ 345us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.912001 (+ 87us) client_negotiation.cc:769] Sending connection context
0810 11:38:01.912130 (+ 129us) client_negotiation.cc:240] Negotiation successful
0810 11:38:01.912414 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20260810 11:38:01.914688 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.900185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47418 (local address 127.3.240.195:33373)
0810 11:38:01.901001 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.901016 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.901053 (+ 37us) server_negotiation.cc:407] Connection header received
0810 11:38:01.901150 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.901158 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.901314 (+ 156us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.901544 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.902631 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.904308 (+ 1677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.908774 (+ 4466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.911191 (+ 2417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.913452 (+ 2261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.913693 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:01.913792 (+ 99us) server_negotiation.cc:299] Negotiation successful
0810 11:38:01.914021 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":462,"threads_started":1}
I20260810 11:38:01.916307 14596 tablet_service.cc:1467] Processing CreateTablet for tablet 3010be78f8f9452fa38062665a9f214b (DEFAULT_TABLE table=test-table [id=984a36aea7514e18a338f348d80d67e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:01.917289 14596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3010be78f8f9452fa38062665a9f214b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:01.920739 14849 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.902539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38681 (local address 127.0.0.1:60402)
0810 11:38:01.905496 (+ 2957us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.905534 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.905699 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.906756 (+ 1057us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.906773 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.906800 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.907708 (+ 908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.907728 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.909986 (+ 2258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.909998 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.916096 (+ 6098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.916118 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.919067 (+ 2949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.919086 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.919441 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.919515 (+ 74us) client_negotiation.cc:769] Sending connection context
0810 11:38:01.919681 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 11:38:01.919998 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2730,"thread_start_us":2446,"threads_started":1}
I20260810 11:38:01.920750 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.903298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60402 (local address 127.3.240.193:38681)
0810 11:38:01.906264 (+ 2966us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.906271 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.906306 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:38:01.906421 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.906426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.906523 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.906724 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.907977 (+ 1253us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.909758 (+ 1781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.916327 (+ 6569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.918311 (+ 1984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.918582 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.918803 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:01.919787 (+ 984us) server_negotiation.cc:299] Negotiation successful
0810 11:38:01.920079 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2683,"thread_start_us":2442,"threads_started":1}
I20260810 11:38:01.922442 14824 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.902069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:43305 (local address 127.0.0.1:39144)
0810 11:38:01.902506 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.902546 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.902771 (+ 225us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.909346 (+ 6575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.909351 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.909414 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.910362 (+ 948us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.910529 (+ 167us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.912279 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.912299 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.914655 (+ 2356us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.914670 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.918290 (+ 3620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.918308 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:01.920752 (+ 2444us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.920835 (+ 83us) client_negotiation.cc:769] Sending connection context
0810 11:38:01.921774 (+ 939us) client_negotiation.cc:240] Negotiation successful
0810 11:38:01.921940 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":216}
I20260810 11:38:01.922706 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.902281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39144 (local address 127.3.240.194:43305)
0810 11:38:01.906474 (+ 4193us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.906500 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.906532 (+ 32us) server_negotiation.cc:407] Connection header received
0810 11:38:01.906631 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.906637 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.906772 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.909316 (+ 2544us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.910760 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.912055 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.915689 (+ 3634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.917903 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.918196 (+ 293us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:01.918421 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:01.921879 (+ 3458us) server_negotiation.cc:299] Negotiation successful
0810 11:38:01.922123 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4000,"thread_start_us":3862,"threads_started":1}
I20260810 11:38:01.922660 14173 tablet_service.cc:1467] Processing CreateTablet for tablet 3010be78f8f9452fa38062665a9f214b (DEFAULT_TABLE table=test-table [id=984a36aea7514e18a338f348d80d67e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:01.923466 14173 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3010be78f8f9452fa38062665a9f214b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:01.923764 14386 tablet_service.cc:1467] Processing CreateTablet for tablet 3010be78f8f9452fa38062665a9f214b (DEFAULT_TABLE table=test-table [id=984a36aea7514e18a338f348d80d67e2]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:01.924698 14386 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3010be78f8f9452fa38062665a9f214b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:01.936975 14852 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Bootstrap starting.
I20260810 11:38:01.940202 14855 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Bootstrap starting.
I20260810 11:38:01.941220 14854 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Bootstrap starting.
I20260810 11:38:01.942749 14855 tablet_bootstrap.cc:654] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:01.944245 14854 tablet_bootstrap.cc:654] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:01.944525 14852 tablet_bootstrap.cc:654] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:01.949966 14855 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: No bootstrap required, opened a new log
I20260810 11:38:01.950255 14855 ts_tablet_manager.cc:1397] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.007s
I20260810 11:38:01.951467 14852 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: No bootstrap required, opened a new log
I20260810 11:38:01.951716 14852 ts_tablet_manager.cc:1397] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260810 11:38:01.951785 14854 tablet_bootstrap.cc:492] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: No bootstrap required, opened a new log
I20260810 11:38:01.952044 14854 ts_tablet_manager.cc:1397] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260810 11:38:01.952000 14855 raft_consensus.cc:348] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.952605 14855 raft_consensus.cc:374] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:01.952786 14855 raft_consensus.cc:729] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9c8090f25aa7434e856fb7949607deb7, State: Initialized, Role: FOLLOWER
I20260810 11:38:01.953191 14852 raft_consensus.cc:348] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.953492 14852 raft_consensus.cc:374] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:01.953365 14855 consensus_queue.cc:260] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.953672 14852 raft_consensus.cc:729] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d812df4cd66145dd8e713375f6ad2e3d, State: Initialized, Role: FOLLOWER
I20260810 11:38:01.953712 14854 raft_consensus.cc:348] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.954190 14854 raft_consensus.cc:374] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:01.954037 14852 consensus_queue.cc:260] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.954425 14854 raft_consensus.cc:729] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6160c92c581a455798298753a797bbfb, State: Initialized, Role: FOLLOWER
I20260810 11:38:01.955036 14854 consensus_queue.cc:260] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.956175 14855 ts_tablet_manager.cc:1428] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Time spent starting tablet: real 0.006s user 0.003s sys 0.003s
I20260810 11:38:01.963145 14865 raft_consensus.cc:482] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:01.964352 14865 raft_consensus.cc:504] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:01.975586 14852 ts_tablet_manager.cc:1428] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Time spent starting tablet: real 0.024s user 0.001s sys 0.004s
I20260810 11:38:01.981921 14865 leader_election.cc:288] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373), 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:38:01.984436 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.956606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:59205)
0810 11:38:01.983670 (+ 27064us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.984003 (+ 333us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26832,"thread_start_us":26595,"threads_started":1}
I20260810 11:38:01.988945 14483 heartbeater.cc:503] Master 127.3.240.252:39553 was elected leader, sending a full tablet report...
I20260810 11:38:01.989162 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.967880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:41361)
0810 11:38:01.988521 (+ 20641us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.988782 (+ 261us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20482,"threads_started":1}
I20260810 11:38:01.992379 14694 heartbeater.cc:503] Master 127.3.240.252:39553 was elected leader, sending a full tablet report...
I20260810 11:38:01.998278 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.993372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:50933)
0810 11:38:01.997993 (+ 4621us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.998098 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4508}
I20260810 11:38:01.998459 14854 ts_tablet_manager.cc:1428] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Time spent starting tablet: real 0.046s user 0.025s sys 0.012s
I20260810 11:38:02.002308 14873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.990198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:38383)
0810 11:38:02.001528 (+ 11330us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.001835 (+ 307us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":11188,"threads_started":1}
I20260810 11:38:02.007097 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.968568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52379 (local address 127.3.240.193:38681)
0810 11:38:01.968835 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.968842 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.987064 (+ 18222us) server_negotiation.cc:407] Connection header received
0810 11:38:01.988337 (+ 1273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.988344 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.988456 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.988541 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:01.990912 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.992545 (+ 1633us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.002603 (+ 10058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.004841 (+ 2238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.005082 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.005289 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.006333 (+ 1044us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.006622 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 11:38:02.007937 14870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.967982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38681 (local address 127.3.240.194:52379)
0810 11:38:01.986735 (+ 18753us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.986770 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.987747 (+ 977us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.988939 (+ 1192us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.988944 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.988977 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:01.989964 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:01.989985 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:01.992880 (+ 2895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:01.992887 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:02.000670 (+ 7783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:02.000684 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.005697 (+ 5013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.005705 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:02.006043 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.006099 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 11:38:02.006304 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 11:38:02.006734 (+ 430us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18589,"thread_start_us":29291,"threads_started":1}
I20260810 11:38:02.011016 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.999228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:35639)
0810 11:38:02.010073 (+ 10845us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.010427 (+ 354us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":10563,"thread_start_us":10086,"threads_started":1}
I20260810 11:38:02.011605 14847 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.987350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:45953 (local address 127.3.240.195:33373)
0810 11:38:01.987697 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:01.987707 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:01.997841 (+ 10134us) server_negotiation.cc:407] Connection header received
0810 11:38:01.998967 (+ 1126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:01.998980 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:01.999069 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:01.999144 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:02.000425 (+ 1281us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.002100 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.006580 (+ 4480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.009277 (+ 2697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.009563 (+ 286us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.009727 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.010832 (+ 1105us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.011197 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260810 11:38:02.011904 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:01.987606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33373 (local address 127.3.240.194:45953)
0810 11:38:01.997708 (+ 10102us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:01.997737 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:01.998715 (+ 978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:01.999385 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:01.999389 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:01.999422 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:02.000004 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:02.000020 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.002405 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.002423 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:02.005904 (+ 3481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:02.005919 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.010073 (+ 4154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.010092 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:02.010426 (+ 334us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.010504 (+ 78us) client_negotiation.cc:769] Sending connection context
0810 11:38:02.011327 (+ 823us) client_negotiation.cc:240] Negotiation successful
0810 11:38:02.011491 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9872}
I20260810 11:38:02.013051 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3010be78f8f9452fa38062665a9f214b" candidate_uuid: "9c8090f25aa7434e856fb7949607deb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6160c92c581a455798298753a797bbfb" is_pre_election: true
I20260810 11:38:02.013406 14203 raft_consensus.cc:2454] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8090f25aa7434e856fb7949607deb7 in term 0.
I20260810 11:38:02.013309 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3010be78f8f9452fa38062665a9f214b" candidate_uuid: "9c8090f25aa7434e856fb7949607deb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812df4cd66145dd8e713375f6ad2e3d" is_pre_election: true
I20260810 11:38:02.013882 14627 raft_consensus.cc:2454] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9c8090f25aa7434e856fb7949607deb7 in term 0.
I20260810 11:38:02.014242 14266 heartbeater.cc:503] Master 127.3.240.252:39553 was elected leader, sending a full tablet report...
I20260810 11:38:02.014855 14320 leader_election.cc:302] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 6160c92c581a455798298753a797bbfb, 9c8090f25aa7434e856fb7949607deb7, d812df4cd66145dd8e713375f6ad2e3d; no voters:
I20260810 11:38:02.015735 14869 raft_consensus.cc:2781] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:02.015959 14869 raft_consensus.cc:482] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:02.016238 14869 raft_consensus.cc:3037] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:02.019662 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.014294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:42019)
0810 11:38:02.019148 (+ 4854us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.019345 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1004}
I20260810 11:38:02.021699 14869 raft_consensus.cc:504] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:02.023137 14869 leader_election.cc:288] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [CANDIDATE]: Term 1 election: Requested vote from peers d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373), 6160c92c581a455798298753a797bbfb (127.3.240.193:38681)
I20260810 11:38:02.023844 14627 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3010be78f8f9452fa38062665a9f214b" candidate_uuid: "9c8090f25aa7434e856fb7949607deb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d812df4cd66145dd8e713375f6ad2e3d"
I20260810 11:38:02.023841 14203 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3010be78f8f9452fa38062665a9f214b" candidate_uuid: "9c8090f25aa7434e856fb7949607deb7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6160c92c581a455798298753a797bbfb"
I20260810 11:38:02.024168 14627 raft_consensus.cc:3037] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:02.024178 14203 raft_consensus.cc:3037] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:02.027595 14203 raft_consensus.cc:2454] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8090f25aa7434e856fb7949607deb7 in term 1.
I20260810 11:38:02.027890 14627 raft_consensus.cc:2454] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9c8090f25aa7434e856fb7949607deb7 in term 1.
I20260810 11:38:02.028443 14315 leader_election.cc:302] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6160c92c581a455798298753a797bbfb, 9c8090f25aa7434e856fb7949607deb7; no voters:
I20260810 11:38:02.029001 14869 raft_consensus.cc:2781] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:02.029421 14869 raft_consensus.cc:686] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 1 LEADER]: Becoming Leader. State: Replica: 9c8090f25aa7434e856fb7949607deb7, State: Running, Role: LEADER
I20260810 11:38:02.030246 14869 consensus_queue.cc:237] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } }
I20260810 11:38:02.031051 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.030283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:58649)
0810 11:38:02.030612 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.030797 (+ 185us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260810 11:38:02.036159 13995 catalog_manager.cc:5495] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "9c8090f25aa7434e856fb7949607deb7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9c8090f25aa7434e856fb7949607deb7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 43305 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:02.095754 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.085051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.3.240.194:43305)
0810 11:38:02.085578 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:02.085583 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:02.085612 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:38:02.085785 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:02.085850 (+ 65us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:02.086111 (+ 261us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:02.086342 (+ 231us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:02.087274 (+ 932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.089092 (+ 1818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.091259 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.091943 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.092134 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.093085 (+ 951us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:02.094625 (+ 1540us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:02.094888 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.095203 (+ 315us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.095328 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129}
I20260810 11:38:02.103471 14203 raft_consensus.cc:1261] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Refusing update from remote peer 9c8090f25aa7434e856fb7949607deb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:02.104136 14627 raft_consensus.cc:1261] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Refusing update from remote peer 9c8090f25aa7434e856fb7949607deb7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:02.104694 14865 consensus_queue.cc:1035] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6160c92c581a455798298753a797bbfb" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38681 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:02.105424 14869 consensus_queue.cc:1035] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d812df4cd66145dd8e713375f6ad2e3d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33373 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:02.111711 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.110369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:44893)
0810 11:38:02.111063 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.111272 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337}
I20260810 11:38:02.112594 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.111795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:50541)
0810 11:38:02.112023 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.112225 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260810 11:38:02.130149 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.125585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:43855)
0810 11:38:02.129745 (+ 4160us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.129913 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":3992}
I20260810 11:38:02.136234 14874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.135050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.193:36635)
0810 11:38:02.135756 (+ 706us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.135911 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":538}
I20260810 11:38:02.141373 14872 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.139528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.194:52521)
0810 11:38:02.140777 (+ 1249us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.141023 (+ 246us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1054}
I20260810 11:38:02.145517 14871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.144345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.3.240.195:39501)
0810 11:38:02.144931 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.145140 (+ 209us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
I20260810 11:38:02.162421 13996 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:02.163022 13996 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37456:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:02.175303 13996 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 970e8a6cd0a840ab99fabcf63b58ee1f: Sending DeleteTablet for 3 replicas of tablet 3010be78f8f9452fa38062665a9f214b
I20260810 11:38:02.177260 14386 tablet_service.cc:1514] Processing DeleteTablet for tablet 3010be78f8f9452fa38062665a9f214b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:02 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:39144
I20260810 11:38:02.177192 14596 tablet_service.cc:1514] Processing DeleteTablet for tablet 3010be78f8f9452fa38062665a9f214b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:02 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:47418
I20260810 11:38:02.178031 14173 tablet_service.cc:1514] Processing DeleteTablet for tablet 3010be78f8f9452fa38062665a9f214b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:02 UTC) from {username='oryx', principal='oryx/127.3.240.252@KRBTEST.COM'} at 127.0.0.1:60402
I20260810 11:38:02.186717 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 13959
I20260810 11:38:02.190354 14888 tablet_replica.cc:330] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: stopping tablet replica
I20260810 11:38:02.191498 14894 tablet_replica.cc:330] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: stopping tablet replica
I20260810 11:38:02.192286 14888 raft_consensus.cc:2229] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:02.194306 14888 raft_consensus.cc:2258] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:02.192960 14894 raft_consensus.cc:2229] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:02.195454 14894 raft_consensus.cc:2258] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:02.196702 14886 tablet_replica.cc:330] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: stopping tablet replica
I20260810 11:38:02.197031 14886 raft_consensus.cc:2229] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:02.197624 14886 raft_consensus.cc:2258] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:02.199187 14888 ts_tablet_manager.cc:1905] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:02.200536 14886 ts_tablet_manager.cc:1905] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:02.207496 14894 ts_tablet_manager.cc:1905] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:02.214221 14888 ts_tablet_manager.cc:1918] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:02.214651 14888 log.cc:1196] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/3010be78f8f9452fa38062665a9f214b
I20260810 11:38:02.215766 14888 ts_tablet_manager.cc:1939] T 3010be78f8f9452fa38062665a9f214b P 9c8090f25aa7434e856fb7949607deb7: Deleting consensus metadata
I20260810 11:38:02.218315 13978 catalog_manager.cc:4841] TS 9c8090f25aa7434e856fb7949607deb7 (127.3.240.194:43305): tablet 3010be78f8f9452fa38062665a9f214b (table test-table [id=984a36aea7514e18a338f348d80d67e2]) successfully deleted
I20260810 11:38:02.221553 14886 ts_tablet_manager.cc:1918] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:02.221786 14886 log.cc:1196] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/3010be78f8f9452fa38062665a9f214b
I20260810 11:38:02.222648 14886 ts_tablet_manager.cc:1939] T 3010be78f8f9452fa38062665a9f214b P d812df4cd66145dd8e713375f6ad2e3d: Deleting consensus metadata
I20260810 11:38:02.225075 13981 catalog_manager.cc:4841] TS d812df4cd66145dd8e713375f6ad2e3d (127.3.240.195:33373): tablet 3010be78f8f9452fa38062665a9f214b (table test-table [id=984a36aea7514e18a338f348d80d67e2]) successfully deleted
I20260810 11:38:02.227813 14894 ts_tablet_manager.cc:1918] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:02.228233 14894 log.cc:1196] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/3010be78f8f9452fa38062665a9f214b
I20260810 11:38:02.228324 14851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.191624 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60410 (local address 127.3.240.193:38681)
0810 11:38:02.191951 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:02.191983 (+ 32us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:02.192759 (+ 776us) server_negotiation.cc:407] Connection header received
0810 11:38:02.193237 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:02.193243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:02.193705 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:02.193908 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:02.198674 (+ 4766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.200504 (+ 1830us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.203998 (+ 3494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.205275 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.205500 (+ 225us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.212552 (+ 7052us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:02.212574 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:02.212609 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:02.212694 (+ 85us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:02.220237 (+ 7543us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.222562 (+ 2325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.222587 (+ 25us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.222606 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.223224 (+ 618us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.224185 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.224190 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.224203 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.225101 (+ 898us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:02.225822 (+ 721us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.226606 (+ 784us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.227069 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260810 11:38:02.229720 14257 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:02.229856 14894 ts_tablet_manager.cc:1939] T 3010be78f8f9452fa38062665a9f214b P 6160c92c581a455798298753a797bbfb: Deleting consensus metadata
I20260810 11:38:02.426106 13976 catalog_manager.cc:4841] TS 6160c92c581a455798298753a797bbfb (127.3.240.193:38681): tablet 3010be78f8f9452fa38062665a9f214b (table test-table [id=984a36aea7514e18a338f348d80d67e2]) successfully deleted
I20260810 11:38:02.426823 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 13959
I20260810 11:38:02.471001 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 14292
I20260810 11:38:02.472241 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.470403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51628)
0810 11:38:02.471701 (+ 1298us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.471909 (+ 208us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":673,"threads_started":1}
I20260810 11:38:02.477617 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.476847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51630)
0810 11:38:02.477116 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.477302 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20260810 11:38:02.497371 14848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.472664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39160 (local address 127.3.240.194:43305)
0810 11:38:02.473260 (+ 596us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:02.473281 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:02.473316 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:38:02.473793 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:02.473803 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:02.473912 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:02.474107 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:02.477180 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.478652 (+ 1472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.480668 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.481359 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.481614 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.486577 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:02.486609 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:02.486629 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:02.486679 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:02.492070 (+ 5391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.493440 (+ 1370us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.493449 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.493451 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.493718 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.494461 (+ 743us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.494465 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.494467 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.495139 (+ 672us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:02.495775 (+ 636us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.496210 (+ 435us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.496611 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373}
I20260810 11:38:02.498575 14475 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:02.670087 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 14292
I20260810 11:38:02.715538 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 14502
I20260810 11:38:02.742564 14901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.716940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47422 (local address 127.3.240.195:33373)
0810 11:38:02.717706 (+ 766us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:02.717714 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:02.717743 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:38:02.717841 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:02.717846 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:02.717976 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:02.718119 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:02.721137 (+ 3018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.722186 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.724617 (+ 2431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:02.725524 (+ 907us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:02.725787 (+ 263us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:02.730715 (+ 4928us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:02.730748 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:02.730779 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:02.730841 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:02.736468 (+ 5627us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.738348 (+ 1880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.738381 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.738384 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.738775 (+ 391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:02.739569 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:02.739573 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:02.739585 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:02.740576 (+ 991us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:02.740979 (+ 403us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:02.741608 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 11:38:02.741989 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":462,"threads_started":1}
I20260810 11:38:02.743832 14684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:02.880107 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 14502
I20260810 11:38:02.913534 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 13515
I20260810 11:38:02.914767 13754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:02.970844 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.970110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51644)
0810 11:38:02.970418 (+ 308us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.970547 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":145}
I20260810 11:38:02.972097 14897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:02.971649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38543 (local address 127.0.0.1:51654)
0810 11:38:02.971885 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:02.971971 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:38543: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 11:38:03.047611 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 13515
I20260810 11:38:03.080354 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 13653
I20260810 11:38:03.081475 14043 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:03.212570 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 13653
2026-08-10T11:38:03Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (12308 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:38:03 dist-test-slave-drl0 krb5kdc[14907](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:38:03 dist-test-slave-drl0 krb5kdc[14907](info): set up 2 sockets
Aug 10 11:38:03 dist-test-slave-drl0 krb5kdc[14907](info): commencing operation
krb5kdc: starting...
W20260810 11:38:04.574525 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.301s user 0.000s sys 0.038s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:38:04 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:38:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
Entry for principal oryx/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab.
I20260810 11:38:04.871451 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42511
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42511
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:05.251894 14922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:05.252198 14922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:05.252290 14922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:05.259470 14922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:05.259573 14922 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260810 11:38:05.259615 14922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:05.259670 14922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:05.259720 14922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:05.259783 14922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:05.270850 14922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42511
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.254.keytab
--principal=oryx/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:05.272298 14922 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:05.273797 14922 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:05.287336 14935 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:05.287451 14934 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:05.287456 14937 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:05.288522 14922 server_base.cc:1034] running on GCE node
I20260810 11:38:05.289237 14922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:05.290720 14922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:05.292068 14922 hybrid_clock.cc:648] HybridClock initialized: now 1786361885291947 us; error 118 us; skew 500 ppm
I20260810 11:38:05.292574 14922 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:05 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361885, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:05.296298 14922 init.cc:377] Logged in from keytab as oryx/127.3.240.254@KRBTEST.COM (short username oryx)
I20260810 11:38:05.298233 14922 webserver.cc:460] Webserver started at http://127.3.240.254:43709/ using document root <none> and password file <none>
I20260810 11:38:05.299070 14922 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:05.299204 14922 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:05.299602 14922 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:05.303650 14922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "eed9d9b83f7a471596f17bb8d7573f0f"
format_stamp: "Formatted at 2026-08-10 11:38:05 on dist-test-slave-drl0"
server_key: "0db210d8be67ffe6bd76efc582031578"
server_key_iv: "c845a17c3d3212a4848ed4296cce016e"
server_key_version: "encryptionkey@0"
I20260810 11:38:05.304785 14922 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "eed9d9b83f7a471596f17bb8d7573f0f"
format_stamp: "Formatted at 2026-08-10 11:38:05 on dist-test-slave-drl0"
server_key: "0db210d8be67ffe6bd76efc582031578"
server_key_iv: "c845a17c3d3212a4848ed4296cce016e"
server_key_version: "encryptionkey@0"
I20260810 11:38:05.311664 14922 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260810 11:38:05.316907 14948 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:05.319114 14922 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.007s sys 0.000s
I20260810 11:38:05.319401 14922 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "eed9d9b83f7a471596f17bb8d7573f0f"
format_stamp: "Formatted at 2026-08-10 11:38:05 on dist-test-slave-drl0"
server_key: "0db210d8be67ffe6bd76efc582031578"
server_key_iv: "c845a17c3d3212a4848ed4296cce016e"
server_key_version: "encryptionkey@0"
I20260810 11:38:05.319742 14922 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:05.352511 14922 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:05.358194 14922 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:05.358618 14922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:05.385756 14922 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42511
I20260810 11:38:05.385804 15041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42511 every 8 connection(s)
I20260810 11:38:05.387243 14922 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:05.388965 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 14922
I20260810 11:38:05.389184 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:38:05.390075 4035 external_mini_cluster.cc:1428] Setting key 27983af2944dd5cc975cc5efa8293f52
I20260810 11:38:05.392196 15043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 11:38:05 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361884, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:05.405608 15043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Bootstrap starting.
I20260810 11:38:05.409922 15043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:05.411389 15043 log.cc:824] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:05.415402 15043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: No bootstrap required, opened a new log
I20260810 11:38:05.416499 15046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:05.392521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52302 (local address 127.3.240.254:42511)
0810 11:38:05.393694 (+ 1173us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:05.393708 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:05.393766 (+ 58us) server_negotiation.cc:407] Connection header received
0810 11:38:05.395656 (+ 1890us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:05.395677 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:05.396064 (+ 387us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:05.396493 (+ 429us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:05.397973 (+ 1480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:05.399348 (+ 1375us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:05.400557 (+ 1209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:05.401241 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:05.401434 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:05.406199 (+ 4765us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:05.406209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:05.406228 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:05.406277 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:05.410445 (+ 4168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:05.412168 (+ 1723us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:05.412175 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:05.412180 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:05.412363 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:05.412954 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:05.412958 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:05.412960 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:05.413716 (+ 756us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:05.414027 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:05.414713 (+ 686us) server_negotiation.cc:299] Negotiation successful
0810 11:38:05.415181 (+ 468us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":361,"threads_started":1}
I20260810 11:38:05.423804 15043 raft_consensus.cc:348] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } }
I20260810 11:38:05.424391 15043 raft_consensus.cc:374] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:05.424566 15043 raft_consensus.cc:729] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eed9d9b83f7a471596f17bb8d7573f0f, State: Initialized, Role: FOLLOWER
I20260810 11:38:05.425715 15043 consensus_queue.cc:260] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } }
I20260810 11:38:05.426095 15043 raft_consensus.cc:388] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:05.426257 15043 raft_consensus.cc:482] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:05.426482 15043 raft_consensus.cc:3037] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:05.429347 15043 raft_consensus.cc:504] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } }
I20260810 11:38:05.429950 15043 leader_election.cc:302] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eed9d9b83f7a471596f17bb8d7573f0f; no voters:
I20260810 11:38:05.430682 15043 leader_election.cc:288] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:38:05.430894 15050 raft_consensus.cc:2781] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:05.432719 15050 raft_consensus.cc:686] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [term 1 LEADER]: Becoming Leader. State: Replica: eed9d9b83f7a471596f17bb8d7573f0f, State: Running, Role: LEADER
I20260810 11:38:05.433545 15050 consensus_queue.cc:237] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } }
I20260810 11:38:05.434084 15043 sys_catalog.cc:564] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:05.438421 15052 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } } }
I20260810 11:38:05.438885 15052 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:05.438395 15055 sys_catalog.cc:455] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader eed9d9b83f7a471596f17bb8d7573f0f. Latest consensus state: current_term: 1 leader_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eed9d9b83f7a471596f17bb8d7573f0f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42511 } } }
I20260810 11:38:05.439424 15055 sys_catalog.cc:458] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:05.442653 15060 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:05.448516 15060 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:05.467242 15060 catalog_manager.cc:1347] Generated new cluster ID: 43bccd4df3d842d9953b8655d17c7404
I20260810 11:38:05.467371 15060 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:05.480100 15060 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:38:05.481406 15060 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:05.495534 15060 catalog_manager.cc:5868] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Generated new TSK 0
I20260810 11:38:05.496589 15060 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
I20260810 11:38:05.575088 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42511
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:05.954150 15081 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:05.954622 15081 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:05.954850 15081 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:05.963575 15081 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:05.963698 15081 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:05.963801 15081 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:38:05.963928 15081 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:05.975401 15081 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:05.976749 15081 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:05.978060 15081 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:05.993577 15089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:05.993669 15093 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:05.993636 15088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:05.994756 15081 server_base.cc:1034] running on GCE node
I20260810 11:38:05.995697 15081 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:05.996995 15081 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:05.998283 15081 hybrid_clock.cc:648] HybridClock initialized: now 1786361885998209 us; error 40 us; skew 500 ppm
I20260810 11:38:05.998634 15081 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:05 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361885, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:06.002279 15081 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260810 11:38:06.004086 15081 webserver.cc:460] Webserver started at http://127.3.240.193:33135/ using document root <none> and password file <none>
I20260810 11:38:06.004829 15081 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:06.004967 15081 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:06.005311 15081 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:06.008358 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "fc781f0fdf8144e0b3b7656f890bea23"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "cfa5f4d0417c9c8e8371742e729e4e0d"
server_key_iv: "960835d0b648152df6825fab4d951dca"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.010684 15081 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "fc781f0fdf8144e0b3b7656f890bea23"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "cfa5f4d0417c9c8e8371742e729e4e0d"
server_key_iv: "960835d0b648152df6825fab4d951dca"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.018232 15081 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.004s
I20260810 11:38:06.022292 15103 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:06.023993 15081 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:38:06.024196 15081 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "fc781f0fdf8144e0b3b7656f890bea23"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "cfa5f4d0417c9c8e8371742e729e4e0d"
server_key_iv: "960835d0b648152df6825fab4d951dca"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.024508 15081 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:06.048784 15081 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:06.053241 15081 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:06.053530 15081 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:06.056666 15081 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:06.056789 15081 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:06.056955 15081 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:06.057041 15081 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:38:06 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361885, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:06.083673 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.057959 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:42799 (local address 127.3.240.254:42511)
0810 11:38:06.058556 (+ 597us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:06.058561 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:06.059760 (+ 1199us) server_negotiation.cc:407] Connection header received
0810 11:38:06.062060 (+ 2300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:06.062066 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:06.062165 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:06.062300 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:06.064934 (+ 2634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.065885 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.069173 (+ 3288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.069735 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.069888 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.075065 (+ 5177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:06.075070 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:06.075074 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:06.075106 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:06.079213 (+ 4107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.080419 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.080426 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.080430 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.080586 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.081583 (+ 997us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.081590 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.081595 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.081985 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:06.082245 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:06.083029 (+ 784us) server_negotiation.cc:299] Negotiation successful
0810 11:38:06.083264 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":292,"threads_started":1}
I20260810 11:38:06.084903 15116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.058672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.193:42799)
0810 11:38:06.059577 (+ 905us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:06.059623 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:06.061742 (+ 2119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:06.062617 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:06.062632 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:06.063623 (+ 991us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:06.064702 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.064738 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.066077 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.066090 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.068950 (+ 2860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.068970 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.070407 (+ 1437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.070421 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.070615 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.071506 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:06.071550 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:06.074792 (+ 3242us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:06.079385 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.079399 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.079427 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.080247 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.080711 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.080720 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.080724 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.081059 (+ 335us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.082193 (+ 1134us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:06.082207 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:06.082605 (+ 398us) client_negotiation.cc:769] Sending connection context
0810 11:38:06.082904 (+ 299us) client_negotiation.cc:240] Negotiation successful
0810 11:38:06.083514 (+ 610us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":398,"threads_started":1}
I20260810 11:38:06.110427 15115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.092544 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:39005 (local address 127.3.240.254:42511)
0810 11:38:06.092705 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:06.092709 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:06.093717 (+ 1008us) server_negotiation.cc:407] Connection header received
0810 11:38:06.093921 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:06.093925 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:06.093988 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:06.094091 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:06.095067 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.095944 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.098487 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.099513 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.099688 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.102058 (+ 2370us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:06.102068 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:06.102077 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:06.102130 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:06.106480 (+ 4350us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.107412 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.107419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.107426 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.107598 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.108306 (+ 708us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.108310 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.108313 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.108870 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:06.109037 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:06.109996 (+ 959us) server_negotiation.cc:299] Negotiation successful
0810 11:38:06.110177 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 11:38:06.111073 15203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.092742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.193:39005)
0810 11:38:06.093602 (+ 860us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:06.093620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:06.093790 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:06.094114 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:06.094118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:06.094603 (+ 485us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:06.094891 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.094901 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.096191 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.096199 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.098232 (+ 2033us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.098255 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.099918 (+ 1663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.099928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.100126 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.101038 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:06.101057 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:06.101797 (+ 740us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:06.106680 (+ 4883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.106689 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.106693 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.107274 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.107762 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.107775 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.107781 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.107999 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.109175 (+ 1176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:06.109184 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:06.109577 (+ 393us) client_negotiation.cc:769] Sending connection context
0810 11:38:06.109872 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 11:38:06.110369 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":586,"threads_started":1}
I20260810 11:38:06.124550 15081 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:42427
I20260810 11:38:06.124658 15270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:42427 every 8 connection(s)
I20260810 11:38:06.126053 15081 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:06.128661 15271 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42511
I20260810 11:38:06.128973 15271 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:06.129174 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 15081
I20260810 11:38:06.129405 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:38:06.129832 15271 heartbeater.cc:511] Master 127.3.240.254:42511 requested a full tablet report, sending...
I20260810 11:38:06.130209 4035 external_mini_cluster.cc:1428] Setting key e58fdefa6b56b6a4a95b5e0458b46427
I20260810 11:38:06.132217 14973 ts_manager.cc:194] Registered new tserver with Master: fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427)
I20260810 11:38:06.134375 14973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:42799
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
I20260810 11:38:06.195036 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42511
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:06.592344 15276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:06.592716 15276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:06.592924 15276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:06.600585 15276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:06.600746 15276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:06.600916 15276 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:38:06.601068 15276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:06.612344 15276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:06.613862 15276 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:06.615458 15276 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:06.633047 15287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:06.633153 15284 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:06.633153 15283 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:06.633773 15276 server_base.cc:1034] running on GCE node
I20260810 11:38:06.634505 15276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:06.635643 15276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:06.636919 15276 hybrid_clock.cc:648] HybridClock initialized: now 1786361886636838 us; error 92 us; skew 500 ppm
I20260810 11:38:06.637271 15276 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:06 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361886, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:06.640916 15276 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260810 11:38:06.642760 15276 webserver.cc:460] Webserver started at http://127.3.240.194:45107/ using document root <none> and password file <none>
I20260810 11:38:06.644320 15276 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:06.644476 15276 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:06.644845 15276 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:06.649569 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "afb47787e46695274f9426f50b83629b"
server_key_iv: "b8f64b8a693732d5b7529c09bb956adb"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.650667 15276 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "afb47787e46695274f9426f50b83629b"
server_key_iv: "b8f64b8a693732d5b7529c09bb956adb"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.657181 15276 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:38:06.661612 15294 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:06.663959 15276 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:06.664273 15276 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
format_stamp: "Formatted at 2026-08-10 11:38:06 on dist-test-slave-drl0"
server_key: "afb47787e46695274f9426f50b83629b"
server_key_iv: "b8f64b8a693732d5b7529c09bb956adb"
server_key_version: "encryptionkey@0"
I20260810 11:38:06.664615 15276 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:06.692723 15276 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:06.697616 15276 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:06.697973 15276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:06.701529 15276 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:06.701695 15276 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 11:38:06.701884 15276 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:06.701970 15276 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:38:06 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361886, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:06.728750 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.702020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:56531 (local address 127.3.240.254:42511)
0810 11:38:06.702628 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:06.702639 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:06.704470 (+ 1831us) server_negotiation.cc:407] Connection header received
0810 11:38:06.706555 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:06.706561 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:06.706653 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:06.706787 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:06.709062 (+ 2275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.710232 (+ 1170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.713923 (+ 3691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.714564 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.714726 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.719811 (+ 5085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:06.719816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:06.719819 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:06.719853 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:06.723477 (+ 3624us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.725587 (+ 2110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.725616 (+ 29us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.725624 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.725913 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.726699 (+ 786us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.726708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.726712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.727119 (+ 407us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:06.727305 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:06.728082 (+ 777us) server_negotiation.cc:299] Negotiation successful
0810 11:38:06.728349 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":305,"threads_started":1}
I20260810 11:38:06.729898 15308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.702651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.194:56531)
0810 11:38:06.704282 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:06.704332 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:06.706273 (+ 1941us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:06.707048 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:06.707057 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:06.707922 (+ 865us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:06.708855 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.708889 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.710446 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.710456 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.713730 (+ 3274us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.713750 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.715158 (+ 1408us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.715171 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.715512 (+ 341us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.716507 (+ 995us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:06.716546 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:06.719578 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:06.723747 (+ 4169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.723769 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.723811 (+ 42us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.725202 (+ 1391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.726100 (+ 898us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.726110 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.726117 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.726489 (+ 372us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.727270 (+ 781us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:06.727292 (+ 22us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:06.727692 (+ 400us) client_negotiation.cc:769] Sending connection context
0810 11:38:06.727983 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 11:38:06.728538 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1139,"thread_start_us":664,"threads_started":1}
I20260810 11:38:06.754083 15403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.737516 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.194:34863)
0810 11:38:06.738708 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:06.738735 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:06.738909 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:06.739292 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:06.739297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:06.739770 (+ 473us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:06.740058 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.740067 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.741087 (+ 1020us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.741092 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.742418 (+ 1326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:06.742450 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.743983 (+ 1533us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.743995 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:06.744338 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.745233 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:06.745271 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:06.746751 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:06.750961 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.750969 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.750976 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.751493 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.751906 (+ 413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:06.751912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:06.751918 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:06.752110 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:06.752976 (+ 866us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:06.752979 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:06.753198 (+ 219us) client_negotiation.cc:769] Sending connection context
0810 11:38:06.753403 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 11:38:06.753757 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1028,"thread_start_us":847,"threads_started":1}
I20260810 11:38:06.754458 15306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:06.737181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34863 (local address 127.3.240.254:42511)
0810 11:38:06.737403 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:06.737409 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:06.738832 (+ 1423us) server_negotiation.cc:407] Connection header received
0810 11:38:06.739089 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:06.739094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:06.739157 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:06.739267 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:06.740202 (+ 935us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.740924 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.742720 (+ 1796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:06.743341 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:06.743779 (+ 438us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:06.747104 (+ 3325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:06.747111 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:06.747114 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:06.747150 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:06.750770 (+ 3620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.751639 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.751648 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.751652 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.751785 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:06.752302 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:06.752307 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:06.752310 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:06.752753 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:06.752921 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:06.753563 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 11:38:06.753930 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 11:38:06.767269 15276 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:44747
I20260810 11:38:06.767303 15470 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:44747 every 8 connection(s)
I20260810 11:38:06.768540 15276 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:06.770906 15471 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42511
I20260810 11:38:06.771173 15471 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:06.771799 15471 heartbeater.cc:511] Master 127.3.240.254:42511 requested a full tablet report, sending...
I20260810 11:38:06.773238 14973 ts_manager.cc:194] Registered new tserver with Master: 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747)
I20260810 11:38:06.774067 14973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:56531
I20260810 11:38:06.777948 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 15276
I20260810 11:38:06.778151 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:38:06.778672 4035 external_mini_cluster.cc:1428] Setting key 859e5dadce4cbf0d65be0cdf21a948b1
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260810 11:38:06.851975 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42511
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:38:07.137547 15271 heartbeater.cc:503] Master 127.3.240.254:42511 was elected leader, sending a full tablet report...
W20260810 11:38:07.221478 15475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:07.221787 15475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:07.221899 15475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:07.232527 15475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:07.232656 15475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:07.232756 15475 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260810 11:38:07.232849 15475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:07.244966 15475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.3.240.254:42511
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:07.246721 15475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:07.248421 15475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:07.264057 15481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:07.264197 15484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:07.264266 15480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:07.265618 15475 server_base.cc:1034] running on GCE node
I20260810 11:38:07.266695 15475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:07.268101 15475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:07.269405 15475 hybrid_clock.cc:648] HybridClock initialized: now 1786361887269318 us; error 48 us; skew 500 ppm
I20260810 11:38:07.269809 15475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:07 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361887, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:07.273600 15475 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260810 11:38:07.275775 15475 webserver.cc:460] Webserver started at http://127.3.240.195:33539/ using document root <none> and password file <none>
I20260810 11:38:07.277093 15475 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:07.277271 15475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:07.277702 15475 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:07.281472 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "01c16e176f13491bb3ca8c4518aaa24b"
format_stamp: "Formatted at 2026-08-10 11:38:07 on dist-test-slave-drl0"
server_key: "f996ed9eeeea5d235907eb77f7166998"
server_key_iv: "2bcab632def68140a2ff6d76b667c648"
server_key_version: "encryptionkey@0"
I20260810 11:38:07.282423 15475 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "01c16e176f13491bb3ca8c4518aaa24b"
format_stamp: "Formatted at 2026-08-10 11:38:07 on dist-test-slave-drl0"
server_key: "f996ed9eeeea5d235907eb77f7166998"
server_key_iv: "2bcab632def68140a2ff6d76b667c648"
server_key_version: "encryptionkey@0"
I20260810 11:38:07.288975 15475 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 11:38:07.293051 15492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:07.294659 15475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:38:07.294867 15475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "01c16e176f13491bb3ca8c4518aaa24b"
format_stamp: "Formatted at 2026-08-10 11:38:07 on dist-test-slave-drl0"
server_key: "f996ed9eeeea5d235907eb77f7166998"
server_key_iv: "2bcab632def68140a2ff6d76b667c648"
server_key_version: "encryptionkey@0"
I20260810 11:38:07.295151 15475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:07.306870 15475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:07.311285 15475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:07.311576 15475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:07.315373 15475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:07.315568 15475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:07.315750 15475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:07.315840 15475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:38:07 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361887, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:07.350646 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:07.316599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:33829 (local address 127.3.240.254:42511)
0810 11:38:07.317259 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:07.317266 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:07.318835 (+ 1569us) server_negotiation.cc:407] Connection header received
0810 11:38:07.322670 (+ 3835us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:07.322680 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:07.322793 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:07.322902 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:07.326080 (+ 3178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.327083 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.329235 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.330220 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.330457 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:07.338516 (+ 8059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:07.338528 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:07.338537 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:07.338610 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:07.344761 (+ 6151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:07.346597 (+ 1836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:07.346606 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:07.346610 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:07.346835 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:07.347728 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:07.347733 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:07.347737 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:07.348200 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:07.348423 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:07.349676 (+ 1253us) server_negotiation.cc:299] Negotiation successful
0810 11:38:07.350029 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":311,"threads_started":1}
I20260810 11:38:07.351637 15507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:07.317414 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.195:33829)
0810 11:38:07.318527 (+ 1113us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:07.318624 (+ 97us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:07.322160 (+ 3536us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:07.323299 (+ 1139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:07.323313 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:07.324394 (+ 1081us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:07.325792 (+ 1398us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:07.325847 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.327265 (+ 1418us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.327270 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:07.329075 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:07.329095 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.330656 (+ 1561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.330668 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:07.331056 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:07.332142 (+ 1086us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:07.332197 (+ 55us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:07.338028 (+ 5831us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:07.344985 (+ 6957us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:07.344999 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:07.345027 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:07.346346 (+ 1319us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:07.347035 (+ 689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:07.347086 (+ 51us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:07.347090 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:07.347528 (+ 438us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:07.348410 (+ 882us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:07.348421 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:07.348855 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 11:38:07.349569 (+ 714us) client_negotiation.cc:240] Negotiation successful
0810 11:38:07.350159 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":410,"threads_started":1}
I20260810 11:38:07.382023 15475 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:45557
I20260810 11:38:07.382072 15667 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:45557 every 8 connection(s)
I20260810 11:38:07.383068 15611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:07.361650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.3.240.195:52935)
0810 11:38:07.364598 (+ 2948us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:07.364616 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:07.364811 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:07.365427 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:07.365430 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:07.365985 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:07.366500 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:07.366515 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.367858 (+ 1343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.367864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:07.369988 (+ 2124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:07.370019 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.371338 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.371345 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:07.371599 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:07.373072 (+ 1473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:07.373096 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:07.374214 (+ 1118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:07.379074 (+ 4860us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:07.379086 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:07.379092 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:07.379768 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:07.380334 (+ 566us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:07.380340 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:07.380343 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:07.380502 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:07.381491 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:07.381497 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:07.381849 (+ 352us) client_negotiation.cc:769] Sending connection context
0810 11:38:07.382067 (+ 218us) client_negotiation.cc:240] Negotiation successful
0810 11:38:07.382584 (+ 517us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2794,"thread_start_us":2589,"threads_started":1}
I20260810 11:38:07.383742 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:07.362065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:52935 (local address 127.3.240.254:42511)
0810 11:38:07.362440 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:07.362445 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:07.364740 (+ 2295us) server_negotiation.cc:407] Connection header received
0810 11:38:07.364997 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:07.365003 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:07.365104 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:07.365217 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:07.366686 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.367683 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.370296 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.371076 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.371263 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:07.374484 (+ 3221us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:07.374495 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:07.374502 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:07.374563 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:07.378805 (+ 4242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:07.379950 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:07.379953 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:07.379956 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:07.380170 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:07.380660 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:07.380667 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:07.380671 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:07.381294 (+ 623us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:07.383124 (+ 1830us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:07.383284 (+ 160us) server_negotiation.cc:299] Negotiation successful
0810 11:38:07.383507 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":246}
I20260810 11:38:07.384243 15475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:07.385510 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 15475
I20260810 11:38:07.385682 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:38:07.386270 4035 external_mini_cluster.cc:1428] Setting key d3bcc7b4c4c07709732dc15ddd3c43b2
I20260810 11:38:07.392825 15668 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42511
I20260810 11:38:07.393079 15668 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:07.393826 15668 heartbeater.cc:511] Master 127.3.240.254:42511 requested a full tablet report, sending...
I20260810 11:38:07.395392 14973 ts_manager.cc:194] Registered new tserver with Master: 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:07.396272 14973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:33829
I20260810 11:38:07.403220 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 10 11:38:07 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 10 11:38:07 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:07 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:07.776765 15471 heartbeater.cc:503] Master 127.3.240.254:42511 was elected leader, sending a full tablet report...
I20260810 11:38:08.398974 15668 heartbeater.cc:503] Master 127.3.240.254:42511 was elected leader, sending a full tablet report...
W20260810 11:38:08.649240 15682 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.179s user 0.000s sys 0.022s
W20260810 11:38:08.650781 15506 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:07.463426 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52310 (local address 127.3.240.254:42511)
0810 11:38:07.463707 (+ 281us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:07.463715 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:07.463991 (+ 276us) server_negotiation.cc:407] Connection header received
0810 11:38:07.464276 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:07.464281 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:07.464353 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:07.464476 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:07.466069 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.466829 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.468238 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:07.468670 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:07.468820 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.650415 (+1181595us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52310: BlockingRecv error: recv got EOF from 127.0.0.1:52310 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 10 11:38:08 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361887, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:08.678742 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.656721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52324 (local address 127.3.240.254:42511)
0810 11:38:08.657134 (+ 413us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.657145 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.657590 (+ 445us) server_negotiation.cc:407] Connection header received
0810 11:38:08.657875 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.657882 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.657974 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.658108 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:08.659747 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.660581 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.662291 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.662778 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.662944 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.671788 (+ 8844us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:08.671795 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:08.671800 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:08.671909 (+ 109us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:08.675692 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:08.676518 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:08.676524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:08.676527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:08.676697 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:08.677236 (+ 539us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:08.677242 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:08.677244 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:08.677623 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:08.677841 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.678274 (+ 433us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.678468 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155}
I20260810 11:38:08.684736 14973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:08.693362 14973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:08.725884 15696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.711064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44747 (local address 127.0.0.1:42262)
0810 11:38:08.712577 (+ 1513us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:08.712639 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:08.712925 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:08.715404 (+ 2479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:08.715413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:08.715474 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:08.716281 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.716299 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.718259 (+ 1960us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.718265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.722814 (+ 4549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.722828 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.724900 (+ 2072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.724906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.725136 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.725204 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 11:38:08.725352 (+ 148us) client_negotiation.cc:240] Negotiation successful
0810 11:38:08.725538 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1071,"mutex_wait_us":124,"spinlock_wait_cycles":2048,"thread_start_us":879,"threads_started":1}
I20260810 11:38:08.726135 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.712039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36330 (local address 127.3.240.195:45557)
0810 11:38:08.713363 (+ 1324us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.713379 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.714117 (+ 738us) server_negotiation.cc:407] Connection header received
0810 11:38:08.714563 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.714574 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.714838 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.715074 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:08.716341 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.718051 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.723235 (+ 5184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.724668 (+ 1433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.724807 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.725027 (+ 220us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.725624 (+ 597us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.725811 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1057,"thread_start_us":837,"threads_started":1}
I20260810 11:38:08.726675 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.711383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42262 (local address 127.3.240.194:44747)
0810 11:38:08.714564 (+ 3181us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.714574 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.714604 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:38:08.714707 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.714714 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.714901 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.715118 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:08.716511 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.718038 (+ 1527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.723099 (+ 5061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.724309 (+ 1210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.724465 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.724701 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.726169 (+ 1468us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.726352 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2976,"thread_start_us":2638,"threads_started":1}
I20260810 11:38:08.726902 15697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.711330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45557 (local address 127.0.0.1:36330)
0810 11:38:08.713974 (+ 2644us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:08.713999 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:08.714306 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:08.715404 (+ 1098us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:08.715413 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:08.715474 (+ 61us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:08.716075 (+ 601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.716098 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.718259 (+ 2161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.718265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.722985 (+ 4720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.722996 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.725183 (+ 2187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.725194 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.725394 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.725427 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:38:08.726345 (+ 918us) client_negotiation.cc:240] Negotiation successful
0810 11:38:08.726482 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2489,"mutex_wait_us":571,"spinlock_wait_cycles":8704,"thread_start_us":2294,"threads_started":1}
I20260810 11:38:08.730522 15701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.715233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.0.0.1:51030)
0810 11:38:08.721343 (+ 6110us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:08.721366 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:08.721518 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:08.722214 (+ 696us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:08.722219 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:08.722247 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:08.722741 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.722762 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.724809 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.724816 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.728256 (+ 3440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.728272 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.729776 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.729782 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.729967 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.729998 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 11:38:08.730104 (+ 106us) client_negotiation.cc:240] Negotiation successful
0810 11:38:08.730223 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5953,"thread_start_us":5696,"threads_started":1}
I20260810 11:38:08.730647 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.715912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51030 (local address 127.3.240.193:42427)
0810 11:38:08.719475 (+ 3563us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.719489 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.721486 (+ 1997us) server_negotiation.cc:407] Connection header received
0810 11:38:08.721701 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.721712 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.721992 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.722198 (+ 206us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:08.723015 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.724606 (+ 1591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.728422 (+ 3816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.729538 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.729691 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.729906 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.730163 (+ 257us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.730348 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3112,"thread_start_us":2831,"threads_started":1}
I20260810 11:38:08.731020 15568 tablet_service.cc:1467] Processing CreateTablet for tablet c56f4e40c1a84916ab03e394deb55541 (DEFAULT_TABLE table=test-table [id=c3715dbae0f64cd68763e9e64e32a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.732785 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56f4e40c1a84916ab03e394deb55541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.733132 15369 tablet_service.cc:1467] Processing CreateTablet for tablet c56f4e40c1a84916ab03e394deb55541 (DEFAULT_TABLE table=test-table [id=c3715dbae0f64cd68763e9e64e32a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.735167 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56f4e40c1a84916ab03e394deb55541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.735983 15172 tablet_service.cc:1467] Processing CreateTablet for tablet c56f4e40c1a84916ab03e394deb55541 (DEFAULT_TABLE table=test-table [id=c3715dbae0f64cd68763e9e64e32a26e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.738158 15172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c56f4e40c1a84916ab03e394deb55541. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.746711 15703 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Bootstrap starting.
I20260810 11:38:08.750365 15703 tablet_bootstrap.cc:654] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.751570 15704 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Bootstrap starting.
I20260810 11:38:08.751869 15703 log.cc:824] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:08.754009 15706 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Bootstrap starting.
I20260810 11:38:08.756136 15704 tablet_bootstrap.cc:654] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.757527 15704 log.cc:824] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:08.758453 15706 tablet_bootstrap.cc:654] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.758589 15703 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: No bootstrap required, opened a new log
I20260810 11:38:08.758870 15703 ts_tablet_manager.cc:1397] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Time spent bootstrapping tablet: real 0.012s user 0.007s sys 0.000s
I20260810 11:38:08.759896 15706 log.cc:824] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:08.761235 15704 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: No bootstrap required, opened a new log
I20260810 11:38:08.761495 15704 ts_tablet_manager.cc:1397] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20260810 11:38:08.764011 15706 tablet_bootstrap.cc:492] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: No bootstrap required, opened a new log
I20260810 11:38:08.764298 15706 ts_tablet_manager.cc:1397] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260810 11:38:08.770504 15703 raft_consensus.cc:348] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.771056 15703 raft_consensus.cc:374] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.771183 15703 raft_consensus.cc:729] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.772133 15703 consensus_queue.cc:260] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.772986 15704 raft_consensus.cc:348] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.773610 15704 raft_consensus.cc:374] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.773767 15704 raft_consensus.cc:729] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.774741 15704 consensus_queue.cc:260] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.777168 15706 raft_consensus.cc:348] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.777597 15706 raft_consensus.cc:374] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.777694 15706 raft_consensus.cc:729] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.778321 15706 consensus_queue.cc:260] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.780831 15704 ts_tablet_manager.cc:1428] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent starting tablet: real 0.019s user 0.010s sys 0.010s
I20260810 11:38:08.783202 15703 ts_tablet_manager.cc:1428] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Time spent starting tablet: real 0.024s user 0.012s sys 0.012s
I20260810 11:38:08.788129 15706 ts_tablet_manager.cc:1428] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Time spent starting tablet: real 0.024s user 0.014s sys 0.007s
I20260810 11:38:08.795919 15713 raft_consensus.cc:482] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:08.796240 15713 raft_consensus.cc:504] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.798514 15713 leader_election.cc:288] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557), fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427)
I20260810 11:38:08.806111 15700 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.798963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:57753 (local address 127.3.240.195:45557)
0810 11:38:08.799168 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.799177 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.800170 (+ 993us) server_negotiation.cc:407] Connection header received
0810 11:38:08.800399 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.800403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.800461 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.800540 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:08.801185 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.802023 (+ 838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.804405 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.805189 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.805379 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.805458 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.805725 (+ 267us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.805861 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260810 11:38:08.806263 15716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.798900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45557 (local address 127.3.240.194:57753)
0810 11:38:08.800074 (+ 1174us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:08.800092 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:08.800225 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:08.800579 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:08.800582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:08.800603 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:08.801012 (+ 409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.801021 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.802184 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.802189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.804245 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.804258 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.805409 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.805415 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.805562 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.805587 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:38:08.805699 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 11:38:08.805838 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":999,"mutex_wait_us":4,"thread_start_us":492,"threads_started":1}
I20260810 11:38:08.806560 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.799391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53229 (local address 127.3.240.193:42427)
0810 11:38:08.799591 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.799597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.799877 (+ 280us) server_negotiation.cc:407] Connection header received
0810 11:38:08.800124 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.800142 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.800230 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.800321 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:08.801459 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.802156 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.804802 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.805572 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.805768 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.805845 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.806159 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.806307 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:38:08.806918 15715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.798828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.3.240.194:53229)
0810 11:38:08.799663 (+ 835us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:08.799683 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:08.799964 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:08.800781 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:08.800784 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:08.800892 (+ 108us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:08.801261 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.801269 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.802408 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.802417 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.804621 (+ 2204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:08.804634 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.805811 (+ 1177us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.805818 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:08.805988 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.806010 (+ 22us) client_negotiation.cc:769] Sending connection context
0810 11:38:08.806123 (+ 113us) client_negotiation.cc:240] Negotiation successful
0810 11:38:08.806535 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"mutex_wait_us":44,"spinlock_wait_cycles":17792,"thread_start_us":766,"threads_started":1}
I20260810 11:38:08.807387 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56f4e40c1a84916ab03e394deb55541" candidate_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b" is_pre_election: true
I20260810 11:38:08.808014 15594 raft_consensus.cc:2454] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea9d9fbfe18400c80b6d2d42a2121c7 in term 0.
I20260810 11:38:08.808562 15209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56f4e40c1a84916ab03e394deb55541" candidate_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc781f0fdf8144e0b3b7656f890bea23" is_pre_election: true
I20260810 11:38:08.808817 15296 leader_election.cc:302] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, 4ea9d9fbfe18400c80b6d2d42a2121c7; no voters:
I20260810 11:38:08.809338 15209 raft_consensus.cc:2454] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4ea9d9fbfe18400c80b6d2d42a2121c7 in term 0.
I20260810 11:38:08.809638 15713 raft_consensus.cc:2781] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:08.809847 15713 raft_consensus.cc:482] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:08.810009 15713 raft_consensus.cc:3037] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:08.812754 15713 raft_consensus.cc:504] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.813462 15713 leader_election.cc:288] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [CANDIDATE]: Term 1 election: Requested vote from peers 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557), fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427)
I20260810 11:38:08.814074 15209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56f4e40c1a84916ab03e394deb55541" candidate_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc781f0fdf8144e0b3b7656f890bea23"
I20260810 11:38:08.814355 15209 raft_consensus.cc:3037] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:08.814368 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c56f4e40c1a84916ab03e394deb55541" candidate_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b"
I20260810 11:38:08.814728 15594 raft_consensus.cc:3037] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:08.817099 15209 raft_consensus.cc:2454] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea9d9fbfe18400c80b6d2d42a2121c7 in term 1.
I20260810 11:38:08.817632 15594 raft_consensus.cc:2454] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4ea9d9fbfe18400c80b6d2d42a2121c7 in term 1.
I20260810 11:38:08.817632 15297 leader_election.cc:302] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea9d9fbfe18400c80b6d2d42a2121c7, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:08.818022 15713 raft_consensus.cc:2781] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:08.818580 15713 raft_consensus.cc:686] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 LEADER]: Becoming Leader. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Running, Role: LEADER
I20260810 11:38:08.819391 15713 consensus_queue.cc:237] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } }
I20260810 11:38:08.825414 14971 catalog_manager.cc:5495] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:08.864248 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:08.857491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42278 (local address 127.3.240.194:44747)
0810 11:38:08.857793 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:08.857801 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:08.857932 (+ 131us) server_negotiation.cc:407] Connection header received
0810 11:38:08.858211 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:08.858216 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:08.858314 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:08.858490 (+ 176us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:08.858899 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.859756 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.861268 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:08.861877 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:08.862018 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:08.862593 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:08.863397 (+ 804us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:08.863513 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:08.863819 (+ 306us) server_negotiation.cc:299] Negotiation successful
0810 11:38:08.863949 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
I20260810 11:38:08.874084 15594 raft_consensus.cc:1261] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Refusing update from remote peer 4ea9d9fbfe18400c80b6d2d42a2121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:08.874089 15209 raft_consensus.cc:1261] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Refusing update from remote peer 4ea9d9fbfe18400c80b6d2d42a2121c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:08.875394 15717 consensus_queue.cc:1035] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:08.875790 15713 consensus_queue.cc:1035] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 11:38:08.880554 15272 tablet.cc:2369] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:38:08.903636 14972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:08.904043 14972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:08.908586 14972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Sending DeleteTablet for 3 replicas of tablet c56f4e40c1a84916ab03e394deb55541
I20260810 11:38:08.909904 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet c56f4e40c1a84916ab03e394deb55541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:08 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42262
I20260810 11:38:08.910512 15568 tablet_service.cc:1514] Processing DeleteTablet for tablet c56f4e40c1a84916ab03e394deb55541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:08 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36330
I20260810 11:38:08.910509 15172 tablet_service.cc:1514] Processing DeleteTablet for tablet c56f4e40c1a84916ab03e394deb55541 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:08 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51030
I20260810 11:38:08.911036 15735 tablet_replica.cc:330] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: stopping tablet replica
I20260810 11:38:08.911684 15735 raft_consensus.cc:2229] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:08.911924 15739 tablet_replica.cc:330] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: stopping tablet replica
I20260810 11:38:08.912392 15739 raft_consensus.cc:2229] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:08.912539 15735 raft_consensus.cc:2258] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:08.913040 15739 raft_consensus.cc:2258] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:08.913861 14971 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:08.914708 14971 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:08.915254 15735 ts_tablet_manager.cc:1905] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:08.919296 15739 ts_tablet_manager.cc:1905] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:08.924562 15568 tablet_service.cc:1467] Processing CreateTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b (DEFAULT_TABLE table=test-table [id=e608083ab344479ca9947891b32203da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.924666 15369 tablet_service.cc:1467] Processing CreateTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b (DEFAULT_TABLE table=test-table [id=e608083ab344479ca9947891b32203da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.925062 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6169c5b0fce4e479e4108cb160d0f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.925150 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6169c5b0fce4e479e4108cb160d0f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.927465 15740 tablet_replica.cc:330] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: stopping tablet replica
I20260810 11:38:08.927945 15740 raft_consensus.cc:2229] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:08.928530 15740 raft_consensus.cc:2258] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:08.928732 15171 tablet_service.cc:1467] Processing CreateTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b (DEFAULT_TABLE table=test-table [id=e608083ab344479ca9947891b32203da]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:08.929313 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a6169c5b0fce4e479e4108cb160d0f8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:08.923708 15735 ts_tablet_manager.cc:1918] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:08.931991 15735 log.cc:1196] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/c56f4e40c1a84916ab03e394deb55541
I20260810 11:38:08.932621 15735 ts_tablet_manager.cc:1939] T c56f4e40c1a84916ab03e394deb55541 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting consensus metadata
I20260810 11:38:08.934301 15703 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Bootstrap starting.
I20260810 11:38:08.934958 15740 ts_tablet_manager.cc:1905] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:08.935417 14955 catalog_manager.cc:4841] TS 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747): tablet c56f4e40c1a84916ab03e394deb55541 (table test-table [id=c3715dbae0f64cd68763e9e64e32a26e]) successfully deleted
I20260810 11:38:08.936992 15703 tablet_bootstrap.cc:654] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.938663 15706 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Bootstrap starting.
I20260810 11:38:08.940390 15706 tablet_bootstrap.cc:654] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.941102 15704 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Bootstrap starting.
I20260810 11:38:08.943336 15704 tablet_bootstrap.cc:654] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:08.950731 15704 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: No bootstrap required, opened a new log
I20260810 11:38:08.950899 15704 ts_tablet_manager.cc:1397] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.000s
I20260810 11:38:08.951073 15739 ts_tablet_manager.cc:1918] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:08.951359 15739 log.cc:1196] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/c56f4e40c1a84916ab03e394deb55541
I20260810 11:38:08.953163 15704 raft_consensus.cc:348] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.953509 15704 raft_consensus.cc:374] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.953653 15704 raft_consensus.cc:729] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.954047 15704 consensus_queue.cc:260] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.954437 15706 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: No bootstrap required, opened a new log
I20260810 11:38:08.954742 15706 ts_tablet_manager.cc:1397] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Time spent bootstrapping tablet: real 0.016s user 0.008s sys 0.006s
I20260810 11:38:08.955026 15740 ts_tablet_manager.cc:1918] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:08.955101 15739 ts_tablet_manager.cc:1939] T c56f4e40c1a84916ab03e394deb55541 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting consensus metadata
I20260810 11:38:08.955236 15740 log.cc:1196] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/c56f4e40c1a84916ab03e394deb55541
I20260810 11:38:08.956009 15740 ts_tablet_manager.cc:1939] T c56f4e40c1a84916ab03e394deb55541 P fc781f0fdf8144e0b3b7656f890bea23: Deleting consensus metadata
I20260810 11:38:08.958314 14953 catalog_manager.cc:4841] TS fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427): tablet c56f4e40c1a84916ab03e394deb55541 (table test-table [id=c3715dbae0f64cd68763e9e64e32a26e]) successfully deleted
I20260810 11:38:08.958372 14950 catalog_manager.cc:4841] TS 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557): tablet c56f4e40c1a84916ab03e394deb55541 (table test-table [id=c3715dbae0f64cd68763e9e64e32a26e]) successfully deleted
I20260810 11:38:08.958851 15704 ts_tablet_manager.cc:1428] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260810 11:38:08.958973 15706 raft_consensus.cc:348] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.959345 15706 raft_consensus.cc:374] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.959452 15706 raft_consensus.cc:729] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.959731 15706 consensus_queue.cc:260] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.959914 15703 tablet_bootstrap.cc:492] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: No bootstrap required, opened a new log
I20260810 11:38:08.960451 15706 ts_tablet_manager.cc:1428] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260810 11:38:08.960723 15703 ts_tablet_manager.cc:1397] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.008s
I20260810 11:38:08.961500 15703 raft_consensus.cc:348] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.961723 15703 raft_consensus.cc:374] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:08.961818 15703 raft_consensus.cc:729] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Initialized, Role: FOLLOWER
I20260810 11:38:08.962085 15703 consensus_queue.cc:260] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:08.962598 15703 ts_tablet_manager.cc:1428] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 11:38:09.013839 15712 raft_consensus.cc:482] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:09.014132 15712 raft_consensus.cc:504] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.016782 15712 leader_election.cc:288] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427), 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747)
W20260810 11:38:09.022387 15472 tablet.cc:2369] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:38:09.026854 15702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.018166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47671 (local address 127.3.240.193:42427)
0810 11:38:09.018424 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.018432 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.019225 (+ 793us) server_negotiation.cc:407] Connection header received
0810 11:38:09.019587 (+ 362us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.019599 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.019679 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.019862 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:09.020708 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.021715 (+ 1007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.024361 (+ 2646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.025677 (+ 1316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.025859 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.025941 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.026412 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.026542 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260810 11:38:09.027102 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.017590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38923 (local address 127.3.240.194:44747)
0810 11:38:09.017865 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.017875 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.019350 (+ 1475us) server_negotiation.cc:407] Connection header received
0810 11:38:09.019523 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.019529 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.019638 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.019864 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:09.020585 (+ 721us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.021935 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.024873 (+ 2938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.025892 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.026053 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.026127 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.026611 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.026751 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:38:09.026901 15746 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.017040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.3.240.195:47671)
0810 11:38:09.019069 (+ 2029us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.019093 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.019287 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.020074 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.020078 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.020098 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:09.020556 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.020569 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.021896 (+ 1327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.021901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.024190 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.024206 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.025976 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.025985 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.026230 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.026255 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.026385 (+ 130us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.026517 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1811,"mutex_wait_us":51,"thread_start_us":1934,"threads_started":1}
I20260810 11:38:09.027688 15745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.017217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44747 (local address 127.3.240.195:38923)
0810 11:38:09.019241 (+ 2024us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.019257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.019372 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.019874 (+ 502us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.019879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.019904 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:09.020339 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.020353 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.022620 (+ 2267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.022626 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.024678 (+ 2052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.024690 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.026120 (+ 1430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.026128 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.026418 (+ 290us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.026453 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.027157 (+ 704us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.027312 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1920,"mutex_wait_us":83,"thread_start_us":1407,"threads_started":1}
I20260810 11:38:09.028093 15209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6169c5b0fce4e479e4108cb160d0f8b" candidate_uuid: "01c16e176f13491bb3ca8c4518aaa24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc781f0fdf8144e0b3b7656f890bea23" is_pre_election: true
I20260810 11:38:09.028384 15209 raft_consensus.cc:2454] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c16e176f13491bb3ca8c4518aaa24b in term 0.
I20260810 11:38:09.028656 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6169c5b0fce4e479e4108cb160d0f8b" candidate_uuid: "01c16e176f13491bb3ca8c4518aaa24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" is_pre_election: true
I20260810 11:38:09.028888 15494 leader_election.cc:302] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:09.029125 15402 raft_consensus.cc:2454] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 01c16e176f13491bb3ca8c4518aaa24b in term 0.
I20260810 11:38:09.029322 15712 raft_consensus.cc:2781] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:09.029454 15712 raft_consensus.cc:482] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:09.029574 15712 raft_consensus.cc:3037] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.032971 15712 raft_consensus.cc:504] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.033658 15712 leader_election.cc:288] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [CANDIDATE]: Term 1 election: Requested vote from peers fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427), 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747)
I20260810 11:38:09.034204 15209 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6169c5b0fce4e479e4108cb160d0f8b" candidate_uuid: "01c16e176f13491bb3ca8c4518aaa24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fc781f0fdf8144e0b3b7656f890bea23"
I20260810 11:38:09.034221 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a6169c5b0fce4e479e4108cb160d0f8b" candidate_uuid: "01c16e176f13491bb3ca8c4518aaa24b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
I20260810 11:38:09.034466 15402 raft_consensus.cc:3037] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.034480 15209 raft_consensus.cc:3037] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.037320 15209 raft_consensus.cc:2454] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c16e176f13491bb3ca8c4518aaa24b in term 1.
I20260810 11:38:09.037985 15494 leader_election.cc:302] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:09.038060 15402 raft_consensus.cc:2454] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 01c16e176f13491bb3ca8c4518aaa24b in term 1.
I20260810 11:38:09.038408 15712 raft_consensus.cc:2781] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:09.039094 15712 raft_consensus.cc:686] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 1 LEADER]: Becoming Leader. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Running, Role: LEADER
I20260810 11:38:09.039594 15712 consensus_queue.cc:237] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.043989 14972 catalog_manager.cc:5495] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b reported cstate change: term changed from 0 to 1, leader changed from <none> to 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195). New cstate: current_term: 1 leader_uuid: "01c16e176f13491bb3ca8c4518aaa24b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } health_report { overall_health: HEALTHY } } }
Aug 10 11:38:09 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361885, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.254@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:09.116250 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.099442 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42511 (local address 127.0.0.1:52326)
0810 11:38:09.100231 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.100249 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.100425 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.100990 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.100994 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.101726 (+ 732us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:09.101993 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.102002 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.103598 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.103602 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.104606 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.104618 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.105629 (+ 1011us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.105634 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.105771 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.106470 (+ 699us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:09.106488 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:09.108766 (+ 2278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:09.113161 (+ 4395us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:09.113174 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:09.113181 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:09.113611 (+ 430us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:09.114304 (+ 693us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:09.114308 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:09.114311 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:09.114468 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:09.115200 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:09.115206 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:09.115385 (+ 179us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.115546 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.115755 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":451,"threads_started":1}
I20260810 11:38:09.116557 15506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.099370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52326 (local address 127.3.240.254:42511)
0810 11:38:09.099875 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.099883 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.100379 (+ 496us) server_negotiation.cc:407] Connection header received
0810 11:38:09.100675 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.100683 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.100790 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.100918 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:09.102166 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.103423 (+ 1257us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.104762 (+ 1339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.105396 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.105540 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.109080 (+ 3540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:09.109094 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:09.109102 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:09.109149 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:09.112940 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:09.113952 (+ 1012us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:09.113960 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:09.113963 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:09.114128 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:09.114598 (+ 470us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:09.114604 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:09.114608 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:09.115036 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:09.115207 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.115678 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.115929 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20260810 11:38:09.122813 14972 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:52326:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260810 11:38:09.124135 14972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:09.134500 15171 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=914141574be04dd6997f8175b7e91266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:38:09.134438 15369 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=914141574be04dd6997f8175b7e91266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:38:09.135430 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:09.136160 15568 tablet_service.cc:1467] Processing CreateTablet for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=914141574be04dd6997f8175b7e91266]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260810 11:38:09.136757 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:09.143404 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5c0af1c3075c42ba98c55ca0d5ffe4bd. 1 dirs total, 0 dirs full, 0 dirs failed
W20260810 11:38:09.144124 15670 tablet.cc:2369] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:38:09.144596 15704 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: Bootstrap starting.
I20260810 11:38:09.147264 15704 tablet_bootstrap.cc:654] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.158890 15703 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: Bootstrap starting.
I20260810 11:38:09.159264 15704 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: No bootstrap required, opened a new log
I20260810 11:38:09.159510 15706 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: Bootstrap starting.
I20260810 11:38:09.159608 15704 ts_tablet_manager.cc:1397] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent bootstrapping tablet: real 0.015s user 0.004s sys 0.009s
I20260810 11:38:09.160671 15704 raft_consensus.cc:348] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.160874 15703 tablet_bootstrap.cc:654] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.160989 15704 raft_consensus.cc:374] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.161078 15704 raft_consensus.cc:729] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.161381 15706 tablet_bootstrap.cc:654] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.161427 15704 consensus_queue.cc:260] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.162111 15729 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.162532 15729 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.164373 15704 ts_tablet_manager.cc:1428] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260810 11:38:09.166054 15703 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: No bootstrap required, opened a new log
I20260810 11:38:09.166227 15703 ts_tablet_manager.cc:1397] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: Time spent bootstrapping tablet: real 0.007s user 0.003s sys 0.003s
I20260810 11:38:09.167023 15703 raft_consensus.cc:348] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.167167 15706 tablet_bootstrap.cc:492] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: No bootstrap required, opened a new log
I20260810 11:38:09.167259 15703 raft_consensus.cc:374] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.167346 15703 raft_consensus.cc:729] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.167348 15706 ts_tablet_manager.cc:1397] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.001s
I20260810 11:38:09.167804 15703 consensus_queue.cc:260] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.168128 15706 raft_consensus.cc:348] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.168366 15706 raft_consensus.cc:374] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.168385 15703 ts_tablet_manager.cc:1428] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260810 11:38:09.168439 15706 raft_consensus.cc:729] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.168587 15712 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.168673 15706 consensus_queue.cc:260] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.168908 15712 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.169461 15706 ts_tablet_manager.cc:1428] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260810 11:38:09.169965 15714 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.171200 15714 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.431981 15714 raft_consensus.cc:482] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:09.432781 15714 raft_consensus.cc:504] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.435463 15714 leader_election.cc:288] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:09.445813 15712 consensus_queue.cc:1035] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:09.475956 15748 consensus_queue.cc:1035] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:09.483409 15698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.435872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50203 (local address 127.3.240.194:44747)
0810 11:38:09.436089 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.436099 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.473582 (+ 37483us) server_negotiation.cc:407] Connection header received
0810 11:38:09.473665 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.473670 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.473778 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.473892 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:09.474847 (+ 955us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.476002 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.479736 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.481519 (+ 1783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.481729 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.481848 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.482692 (+ 844us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.482921 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:38:09.483436 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.435852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56715 (local address 127.3.240.195:45557)
0810 11:38:09.469567 (+ 33715us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.469589 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.474238 (+ 4649us) server_negotiation.cc:407] Connection header received
0810 11:38:09.474512 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.474518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.474600 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.474671 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:09.475595 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.476752 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.480301 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.481708 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.481899 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.481986 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.482842 (+ 856us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.483066 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33399,"thread_start_us":33762,"threads_started":1}
I20260810 11:38:09.483769 15767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.436394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45557 (local address 127.3.240.193:56715)
0810 11:38:09.474146 (+ 37752us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.474164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.474334 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.474922 (+ 588us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.474963 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.474991 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:09.475412 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.475425 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.476944 (+ 1519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.476951 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.479931 (+ 2980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.479968 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.482297 (+ 2329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.482305 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.482584 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.482633 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.483027 (+ 394us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.483265 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37635,"mutex_wait_us":26,"thread_start_us":37226,"threads_started":1}
I20260810 11:38:09.484674 15768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.435919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44747 (local address 127.3.240.193:50203)
0810 11:38:09.473330 (+ 37411us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.473353 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.473546 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.474313 (+ 767us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.474318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.474339 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:09.474651 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.474665 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.476623 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.476630 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.479474 (+ 2844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.479500 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.482106 (+ 2606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.482116 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.482421 (+ 305us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.482486 (+ 65us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.483658 (+ 1172us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.483832 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37283,"mutex_wait_us":70,"thread_start_us":39202,"threads_started":1}
I20260810 11:38:09.484831 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0af1c3075c42ba98c55ca0d5ffe4bd" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b" is_pre_election: true
I20260810 11:38:09.485159 15594 raft_consensus.cc:2454] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:09.486231 15104 leader_election.cc:302] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:09.486142 15398 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0af1c3075c42ba98c55ca0d5ffe4bd" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" is_pre_election: true
I20260810 11:38:09.486573 15398 raft_consensus.cc:2454] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:09.486754 15714 raft_consensus.cc:2781] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:09.486896 15714 raft_consensus.cc:482] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:09.487027 15714 raft_consensus.cc:3037] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.489876 15714 raft_consensus.cc:504] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.490520 15714 leader_election.cc:288] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:09.491124 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0af1c3075c42ba98c55ca0d5ffe4bd" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
I20260810 11:38:09.491285 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5c0af1c3075c42ba98c55ca0d5ffe4bd" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b"
I20260810 11:38:09.491425 15402 raft_consensus.cc:3037] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.491513 15594 raft_consensus.cc:3037] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:09.494629 15402 raft_consensus.cc:2454] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:09.494643 15594 raft_consensus.cc:2454] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:09.495150 15104 leader_election.cc:302] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:09.495564 15714 raft_consensus.cc:2781] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:09.496151 15714 raft_consensus.cc:686] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [term 1 LEADER]: Becoming Leader. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Running, Role: LEADER
I20260810 11:38:09.496680 15714 consensus_queue.cc:237] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.498692 15779 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: TxnStatusTablet state changed. Reason: New leader fc781f0fdf8144e0b3b7656f890bea23. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.499315 15779 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:38:09.500241 15782 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260810 11:38:09.500797 15782 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260810 11:38:09.501094 14971 catalog_manager.cc:5495] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:09.550827 15758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.542911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.0.0.1:51042)
0810 11:38:09.543194 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.543221 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.543427 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.544249 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.544256 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.544273 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:38:09.544593 (+ 320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.544605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.546372 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.546377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.547566 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.547580 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.548821 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.548827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.548990 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.549055 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.550341 (+ 1286us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.550362 (+ 21us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.550444 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.550570 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 11:38:09.551179 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.543043 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51042 (local address 127.3.240.193:42427)
0810 11:38:09.543630 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.543636 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.543659 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:38:09.543784 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.543789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.543916 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.544106 (+ 190us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:09.544903 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.546182 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.547727 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.548510 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.548651 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.549330 (+ 679us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.550153 (+ 823us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.550349 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.550521 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.550619 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":450,"thread_start_us":284,"threads_started":1}
I20260810 11:38:09.556490 15594 raft_consensus.cc:1261] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:09.556529 15402 raft_consensus.cc:1261] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:09.557168 15714 consensus_queue.cc:1035] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:09.557514 15779 consensus_queue.cc:1035] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:09.561621 15729 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: TxnStatusTablet state changed. Reason: New leader fc781f0fdf8144e0b3b7656f890bea23. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.562013 15729 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.565721 15772 mvcc.cc:205] Tried to move back new op lower bound from 7316938299615207424 to 7316938299381972992. Current Snapshot: MvccSnapshot[applied={T|T < 7316938299615207424}]
I20260810 11:38:09.572343 15748 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: TxnStatusTablet state changed. Reason: New leader fc781f0fdf8144e0b3b7656f890bea23. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.572669 15748 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.582425 15779 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.582978 15779 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:38:09.587818 15779 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.588258 15779 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P fc781f0fdf8144e0b3b7656f890bea23: This TxnStatusTablet replica's current role is: LEADER
I20260810 11:38:09.592816 15729 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.593111 15729 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 4ea9d9fbfe18400c80b6d2d42a2121c7: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.593828 15748 tablet_replica.cc:439] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } } }
I20260810 11:38:09.594144 15748 tablet_replica.cc:442] T 5c0af1c3075c42ba98c55ca0d5ffe4bd P 01c16e176f13491bb3ca8c4518aaa24b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260810 11:38:09.610224 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.603581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36332 (local address 127.3.240.195:45557)
0810 11:38:09.603775 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.603779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.604429 (+ 650us) server_negotiation.cc:407] Connection header received
0810 11:38:09.604687 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.604691 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.604779 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.604940 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:09.605459 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.606220 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.607788 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.608363 (+ 575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.608530 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.609004 (+ 474us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.609565 (+ 561us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.609723 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.609830 (+ 107us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.609927 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260810 11:38:09.629760 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.622084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57101 (local address 127.3.240.193:42427)
0810 11:38:09.622285 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.622292 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.622549 (+ 257us) server_negotiation.cc:407] Connection header received
0810 11:38:09.622711 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.622717 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.622808 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.622920 (+ 112us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:09.623733 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.625164 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.626831 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.627598 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.627749 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.628680 (+ 931us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.628791 (+ 111us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.628919 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.629085 (+ 166us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.629170 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 11:38:09.629789 15806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.621734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.3.240.195:57101)
0810 11:38:09.622437 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.622454 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.622579 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.623112 (+ 533us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.623117 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.623135 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:38:09.623481 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.623492 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.625401 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.625410 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.626652 (+ 1242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.626667 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.627858 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.627864 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.628074 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.628160 (+ 86us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.628923 (+ 763us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.628940 (+ 17us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.629060 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.629221 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":316,"threads_started":1}
I20260810 11:38:09.691373 15765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.671719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55269 (local address 127.3.240.195:45557)
0810 11:38:09.671909 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.671917 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.684354 (+ 12437us) server_negotiation.cc:407] Connection header received
0810 11:38:09.684688 (+ 334us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.684694 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.684802 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.684928 (+ 126us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:09.685552 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.686444 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.689043 (+ 2599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.689782 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.689926 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.690397 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.690548 (+ 151us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.690645 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.690909 (+ 264us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.691041 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:38:09.691409 15811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.671251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:45557 (local address 127.3.240.193:55269)
0810 11:38:09.684192 (+ 12941us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:09.684217 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:09.684406 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:09.684983 (+ 577us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:09.684989 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:09.685003 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:38:09.685338 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.685351 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.686898 (+ 1547us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.686903 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.688871 (+ 1968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:09.688887 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.689974 (+ 1087us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.689979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:09.690135 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.690200 (+ 65us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:38:09.690698 (+ 498us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:38:09.690710 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 11:38:09.690882 (+ 172us) client_negotiation.cc:240] Negotiation successful
0810 11:38:09.691084 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12451,"thread_start_us":13274,"threads_started":1}
I20260810 11:38:09.731827 14973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:09.732127 14973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:52324:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:09.735468 14973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Sending DeleteTablet for 3 replicas of tablet a6169c5b0fce4e479e4108cb160d0f8b
I20260810 11:38:09.736472 15171 tablet_service.cc:1514] Processing DeleteTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:09 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51030
I20260810 11:38:09.737038 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:09 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42262
I20260810 11:38:09.737573 15568 tablet_service.cc:1514] Processing DeleteTablet for tablet a6169c5b0fce4e479e4108cb160d0f8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:09 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36330
I20260810 11:38:09.762585 15820 tablet_replica.cc:330] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: stopping tablet replica
I20260810 11:38:09.763386 15819 tablet_replica.cc:330] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: stopping tablet replica
I20260810 11:38:09.763790 15820 raft_consensus.cc:2229] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:09.764369 15819 raft_consensus.cc:2229] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:09.765331 15820 raft_consensus.cc:2258] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:09.765775 15819 raft_consensus.cc:2258] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:09.767256 15819 ts_tablet_manager.cc:1905] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:09.767468 15817 tablet_replica.cc:330] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: stopping tablet replica
I20260810 11:38:09.767694 15817 raft_consensus.cc:2229] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:09.768003 15817 raft_consensus.cc:2258] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:09.768492 15820 ts_tablet_manager.cc:1905] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:09.769361 15817 ts_tablet_manager.cc:1905] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:09.777628 15817 ts_tablet_manager.cc:1918] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:09.777804 15817 log.cc:1196] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/a6169c5b0fce4e479e4108cb160d0f8b
I20260810 11:38:09.778373 15817 ts_tablet_manager.cc:1939] T a6169c5b0fce4e479e4108cb160d0f8b P fc781f0fdf8144e0b3b7656f890bea23: Deleting consensus metadata
I20260810 11:38:09.782650 15819 ts_tablet_manager.cc:1918] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:09.782833 14953 catalog_manager.cc:4841] TS fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427): tablet a6169c5b0fce4e479e4108cb160d0f8b (table test-table [id=e608083ab344479ca9947891b32203da]) successfully deleted
I20260810 11:38:09.785288 15820 ts_tablet_manager.cc:1918] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:09.785498 15820 log.cc:1196] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/a6169c5b0fce4e479e4108cb160d0f8b
I20260810 11:38:09.785689 15819 log.cc:1196] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/a6169c5b0fce4e479e4108cb160d0f8b
I20260810 11:38:09.786257 15820 ts_tablet_manager.cc:1939] T a6169c5b0fce4e479e4108cb160d0f8b P 01c16e176f13491bb3ca8c4518aaa24b: Deleting consensus metadata
I20260810 11:38:09.786765 15819 ts_tablet_manager.cc:1939] T a6169c5b0fce4e479e4108cb160d0f8b P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting consensus metadata
I20260810 11:38:09.788317 14955 catalog_manager.cc:4841] TS 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747): tablet a6169c5b0fce4e479e4108cb160d0f8b (table test-table [id=e608083ab344479ca9947891b32203da]) successfully deleted
I20260810 11:38:09.789510 14950 catalog_manager.cc:4841] TS 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557): tablet a6169c5b0fce4e479e4108cb160d0f8b (table test-table [id=e608083ab344479ca9947891b32203da]) successfully deleted
Aug 10 11:38:09 dist-test-slave-drl0 krb5kdc[14907](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 10 11:38:09 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:09 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 11:38:09.873080 15837 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:09.844867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.3.240.254:42511)
0810 11:38:09.855169 (+ 10302us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.855177 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.855657 (+ 480us) server_negotiation.cc:407] Connection header received
0810 11:38:09.855878 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.855885 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.855991 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.856112 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:09.858378 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.860218 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.863393 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.864340 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.864501 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.872710 (+ 8209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:52330: BlockingRecv error: recv got EOF from 127.0.0.1:52330 (error 108)
Metrics: {"server-negotiator.queue_time_us":10099,"thread_start_us":10317,"threads_started":1}
Aug 10 11:38:09 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM
I20260810 11:38:09.904675 15837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:09.880985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52340 (local address 127.3.240.254:42511)
0810 11:38:09.881181 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:09.881188 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:09.883561 (+ 2373us) server_negotiation.cc:407] Connection header received
0810 11:38:09.883771 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:09.883778 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:09.883870 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:09.883987 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:09.886331 (+ 2344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.887641 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.889736 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:09.890403 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:09.890541 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:09.896467 (+ 5926us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:09.896474 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:09.896478 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:09.896525 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:09.900548 (+ 4023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:09.901852 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:09.901858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:09.901862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:09.902054 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:09.902756 (+ 702us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:09.902764 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:09.902770 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:09.903217 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:09.903499 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:09.904018 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 11:38:09.904281 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:38:09.910514 14973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:09.911355 14973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:09.920650 15568 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef0d92bd69e4c7da930930593110581 (DEFAULT_TABLE table=test-table [id=48f151227c814746a9a56564114b2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:09.920732 15171 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef0d92bd69e4c7da930930593110581 (DEFAULT_TABLE table=test-table [id=48f151227c814746a9a56564114b2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:09.921232 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef0d92bd69e4c7da930930593110581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:09.921332 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef0d92bd69e4c7da930930593110581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:09.923521 15369 tablet_service.cc:1467] Processing CreateTablet for tablet 4ef0d92bd69e4c7da930930593110581 (DEFAULT_TABLE table=test-table [id=48f151227c814746a9a56564114b2273]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:09.923980 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4ef0d92bd69e4c7da930930593110581. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:09.936539 15843 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Bootstrap starting.
I20260810 11:38:09.939767 15843 tablet_bootstrap.cc:654] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.947598 15845 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Bootstrap starting.
I20260810 11:38:09.950094 15843 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: No bootstrap required, opened a new log
I20260810 11:38:09.950232 15845 tablet_bootstrap.cc:654] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.950603 15843 ts_tablet_manager.cc:1397] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.004s
I20260810 11:38:09.951727 15843 raft_consensus.cc:348] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.951982 15847 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Bootstrap starting.
I20260810 11:38:09.952152 15843 raft_consensus.cc:374] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.952327 15843 raft_consensus.cc:729] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.952752 15843 consensus_queue.cc:260] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.953756 15843 ts_tablet_manager.cc:1428] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260810 11:38:09.955451 15847 tablet_bootstrap.cc:654] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:09.961045 15845 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: No bootstrap required, opened a new log
I20260810 11:38:09.961366 15845 ts_tablet_manager.cc:1397] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.005s
I20260810 11:38:09.962576 15845 raft_consensus.cc:348] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.963058 15845 raft_consensus.cc:374] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.963238 15845 raft_consensus.cc:729] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.963769 15845 consensus_queue.cc:260] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.965173 15847 tablet_bootstrap.cc:492] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: No bootstrap required, opened a new log
I20260810 11:38:09.965351 15847 ts_tablet_manager.cc:1397] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.007s
I20260810 11:38:09.966444 15847 raft_consensus.cc:348] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.967088 15847 raft_consensus.cc:374] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:09.967352 15847 raft_consensus.cc:729] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Initialized, Role: FOLLOWER
I20260810 11:38:09.967780 15847 consensus_queue.cc:260] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:09.969164 15845 ts_tablet_manager.cc:1428] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260810 11:38:09.971182 15847 ts_tablet_manager.cc:1428] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Time spent starting tablet: real 0.006s user 0.000s sys 0.004s
I20260810 11:38:10.044230 15779 raft_consensus.cc:482] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:10.044497 15779 raft_consensus.cc:504] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.045297 15779 leader_election.cc:288] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:10.045987 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef0d92bd69e4c7da930930593110581" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" is_pre_election: true
I20260810 11:38:10.046271 15402 raft_consensus.cc:2454] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:10.046046 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef0d92bd69e4c7da930930593110581" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b" is_pre_election: true
I20260810 11:38:10.046479 15594 raft_consensus.cc:2454] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:10.046972 15106 leader_election.cc:302] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea9d9fbfe18400c80b6d2d42a2121c7, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:10.047509 15779 raft_consensus.cc:2781] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:10.047709 15779 raft_consensus.cc:482] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:10.047808 15779 raft_consensus.cc:3037] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.051159 15779 raft_consensus.cc:504] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.051926 15779 leader_election.cc:288] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:10.052426 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef0d92bd69e4c7da930930593110581" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
I20260810 11:38:10.052665 15402 raft_consensus.cc:3037] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.052712 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4ef0d92bd69e4c7da930930593110581" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b"
I20260810 11:38:10.052938 15594 raft_consensus.cc:3037] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.055712 15402 raft_consensus.cc:2454] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:10.056198 15106 leader_election.cc:302] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea9d9fbfe18400c80b6d2d42a2121c7, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:10.056233 15594 raft_consensus.cc:2454] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:10.056566 15779 raft_consensus.cc:2781] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:10.057104 15779 raft_consensus.cc:686] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 LEADER]: Becoming Leader. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Running, Role: LEADER
I20260810 11:38:10.057399 15779 consensus_queue.cc:237] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.061812 14971 catalog_manager.cc:5495] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:10.104805 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.098432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51048 (local address 127.3.240.193:42427)
0810 11:38:10.098646 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:10.098653 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:10.099085 (+ 432us) server_negotiation.cc:407] Connection header received
0810 11:38:10.099331 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:10.099336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:10.099427 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:10.099535 (+ 108us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:10.100085 (+ 550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.101178 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.102795 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.103477 (+ 682us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.103614 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.104045 (+ 431us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:10.104188 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:10.104295 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:10.104413 (+ 118us) server_negotiation.cc:299] Negotiation successful
0810 11:38:10.104498 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 11:38:10.108503 15594 raft_consensus.cc:1261] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:10.108527 15402 raft_consensus.cc:1261] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:10.109174 15779 consensus_queue.cc:1035] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:10.109541 15714 consensus_queue.cc:1035] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:10.117070 15771 mvcc.cc:205] Tried to move back new op lower bound from 7316938301877678080 to 7316938301677727744. Current Snapshot: MvccSnapshot[applied={T|T < 7316938301877678080}]
I20260810 11:38:10.127374 14973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:10.128027 14973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:10.137060 14973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Sending DeleteTablet for 3 replicas of tablet 4ef0d92bd69e4c7da930930593110581
I20260810 11:38:10.137835 15171 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef0d92bd69e4c7da930930593110581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51030
I20260810 11:38:10.138159 15817 tablet_replica.cc:330] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: stopping tablet replica
I20260810 11:38:10.138417 15817 raft_consensus.cc:2229] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:10.139008 15817 raft_consensus.cc:2258] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.139053 15568 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef0d92bd69e4c7da930930593110581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36330
I20260810 11:38:10.139348 15820 tablet_replica.cc:330] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: stopping tablet replica
I20260810 11:38:10.139567 15820 raft_consensus.cc:2229] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:10.139791 15820 raft_consensus.cc:2258] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.139798 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet 4ef0d92bd69e4c7da930930593110581 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42262
I20260810 11:38:10.140115 15819 tablet_replica.cc:330] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: stopping tablet replica
I20260810 11:38:10.140331 15819 raft_consensus.cc:2229] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:10.140578 15819 raft_consensus.cc:2258] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.141074 15820 ts_tablet_manager.cc:1905] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.142166 15819 ts_tablet_manager.cc:1905] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.146375 15817 ts_tablet_manager.cc:1905] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.147240 14972 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:10.147975 14972 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:10.150645 15820 ts_tablet_manager.cc:1918] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:10.150795 15820 log.cc:1196] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/4ef0d92bd69e4c7da930930593110581
I20260810 11:38:10.151245 15820 ts_tablet_manager.cc:1939] T 4ef0d92bd69e4c7da930930593110581 P 01c16e176f13491bb3ca8c4518aaa24b: Deleting consensus metadata
I20260810 11:38:10.152381 14950 catalog_manager.cc:4841] TS 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557): tablet 4ef0d92bd69e4c7da930930593110581 (table test-table [id=48f151227c814746a9a56564114b2273]) successfully deleted
I20260810 11:38:10.157999 15171 tablet_service.cc:1467] Processing CreateTablet for tablet debb8b729ef5432aa73e235d3b8a33ab (DEFAULT_TABLE table=test-table [id=f511b43085014df4a610e7b2c1ea8a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:10.158509 15171 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb8b729ef5432aa73e235d3b8a33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:10.163333 15568 tablet_service.cc:1467] Processing CreateTablet for tablet debb8b729ef5432aa73e235d3b8a33ab (DEFAULT_TABLE table=test-table [id=f511b43085014df4a610e7b2c1ea8a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:10.163832 15568 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb8b729ef5432aa73e235d3b8a33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:10.167616 15369 tablet_service.cc:1467] Processing CreateTablet for tablet debb8b729ef5432aa73e235d3b8a33ab (DEFAULT_TABLE table=test-table [id=f511b43085014df4a610e7b2c1ea8a10]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:10.168118 15369 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet debb8b729ef5432aa73e235d3b8a33ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:10.179008 15843 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Bootstrap starting.
I20260810 11:38:10.181103 15843 tablet_bootstrap.cc:654] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:10.191434 15845 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Bootstrap starting.
I20260810 11:38:10.193172 15845 tablet_bootstrap.cc:654] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:10.197322 15847 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Bootstrap starting.
I20260810 11:38:10.200176 15817 ts_tablet_manager.cc:1918] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:10.200377 15817 log.cc:1196] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/4ef0d92bd69e4c7da930930593110581
I20260810 11:38:10.200937 15817 ts_tablet_manager.cc:1939] T 4ef0d92bd69e4c7da930930593110581 P fc781f0fdf8144e0b3b7656f890bea23: Deleting consensus metadata
I20260810 11:38:10.201093 15847 tablet_bootstrap.cc:654] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:10.202497 14953 catalog_manager.cc:4841] TS fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427): tablet 4ef0d92bd69e4c7da930930593110581 (table test-table [id=48f151227c814746a9a56564114b2273]) successfully deleted
I20260810 11:38:10.203404 15819 ts_tablet_manager.cc:1918] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:10.203632 15819 log.cc:1196] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/4ef0d92bd69e4c7da930930593110581
I20260810 11:38:10.204154 15819 ts_tablet_manager.cc:1939] T 4ef0d92bd69e4c7da930930593110581 P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting consensus metadata
I20260810 11:38:10.205472 14955 catalog_manager.cc:4841] TS 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747): tablet 4ef0d92bd69e4c7da930930593110581 (table test-table [id=48f151227c814746a9a56564114b2273]) successfully deleted
I20260810 11:38:10.207610 15843 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: No bootstrap required, opened a new log
I20260810 11:38:10.207828 15843 ts_tablet_manager.cc:1397] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Time spent bootstrapping tablet: real 0.029s user 0.008s sys 0.005s
I20260810 11:38:10.208678 15843 raft_consensus.cc:348] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.208992 15843 raft_consensus.cc:374] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:10.209092 15843 raft_consensus.cc:729] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01c16e176f13491bb3ca8c4518aaa24b, State: Initialized, Role: FOLLOWER
I20260810 11:38:10.209357 15843 consensus_queue.cc:260] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.210868 15845 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: No bootstrap required, opened a new log
I20260810 11:38:10.211072 15845 ts_tablet_manager.cc:1397] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.006s
I20260810 11:38:10.211895 15845 raft_consensus.cc:348] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.212148 15845 raft_consensus.cc:374] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:10.212230 15845 raft_consensus.cc:729] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ea9d9fbfe18400c80b6d2d42a2121c7, State: Initialized, Role: FOLLOWER
I20260810 11:38:10.212628 15845 consensus_queue.cc:260] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.213270 15845 ts_tablet_manager.cc:1428] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260810 11:38:10.215014 15843 ts_tablet_manager.cc:1428] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260810 11:38:10.215540 15847 tablet_bootstrap.cc:492] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: No bootstrap required, opened a new log
I20260810 11:38:10.215744 15847 ts_tablet_manager.cc:1397] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.012s
I20260810 11:38:10.218807 15847 raft_consensus.cc:348] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.219142 15847 raft_consensus.cc:374] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:10.219247 15847 raft_consensus.cc:729] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Initialized, Role: FOLLOWER
I20260810 11:38:10.219525 15847 consensus_queue.cc:260] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.220201 15847 ts_tablet_manager.cc:1428] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260810 11:38:10.345248 15779 raft_consensus.cc:482] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:10.345516 15779 raft_consensus.cc:504] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.346339 15779 leader_election.cc:288] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:10.347155 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb8b729ef5432aa73e235d3b8a33ab" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" is_pre_election: true
I20260810 11:38:10.347172 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb8b729ef5432aa73e235d3b8a33ab" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b" is_pre_election: true
I20260810 11:38:10.347440 15594 raft_consensus.cc:2454] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:10.347445 15402 raft_consensus.cc:2454] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 0.
I20260810 11:38:10.347934 15104 leader_election.cc:302] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 01c16e176f13491bb3ca8c4518aaa24b, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:10.348310 15779 raft_consensus.cc:2781] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:10.348423 15779 raft_consensus.cc:482] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:10.348531 15779 raft_consensus.cc:3037] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.351907 15779 raft_consensus.cc:504] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.352944 15402 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb8b729ef5432aa73e235d3b8a33ab" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7"
I20260810 11:38:10.353165 15402 raft_consensus.cc:3037] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.355294 15594 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "debb8b729ef5432aa73e235d3b8a33ab" candidate_uuid: "fc781f0fdf8144e0b3b7656f890bea23" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "01c16e176f13491bb3ca8c4518aaa24b"
I20260810 11:38:10.355540 15594 raft_consensus.cc:3037] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:10.356289 15402 raft_consensus.cc:2454] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:10.356812 15106 leader_election.cc:302] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4ea9d9fbfe18400c80b6d2d42a2121c7, fc781f0fdf8144e0b3b7656f890bea23; no voters:
I20260810 11:38:10.357097 15714 raft_consensus.cc:2781] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:10.357270 15714 raft_consensus.cc:686] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 1 LEADER]: Becoming Leader. State: Replica: fc781f0fdf8144e0b3b7656f890bea23, State: Running, Role: LEADER
I20260810 11:38:10.357533 15714 consensus_queue.cc:237] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } }
I20260810 11:38:10.358913 15594 raft_consensus.cc:2454] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fc781f0fdf8144e0b3b7656f890bea23 in term 1.
I20260810 11:38:10.360143 15779 leader_election.cc:288] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [CANDIDATE]: Term 1 election: Requested vote from peers 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747), 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557)
I20260810 11:38:10.362474 14972 catalog_manager.cc:5495] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 reported cstate change: term changed from 0 to 1, leader changed from <none> to fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "fc781f0fdf8144e0b3b7656f890bea23" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fc781f0fdf8144e0b3b7656f890bea23" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 42427 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:10.429426 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.420215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56855 (local address 127.3.240.193:42427)
0810 11:38:10.420463 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:10.420473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:10.421868 (+ 1395us) server_negotiation.cc:407] Connection header received
0810 11:38:10.422089 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:10.422094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:10.422172 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:10.422288 (+ 116us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:10.422854 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.423629 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.425619 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.427123 (+ 1504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.427669 (+ 546us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.428287 (+ 618us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:10.428504 (+ 217us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:10.428685 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:10.428745 (+ 60us) server_negotiation.cc:299] Negotiation successful
0810 11:38:10.428845 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":59}
I20260810 11:38:10.429651 15864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.420214 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:42427 (local address 127.3.240.193:56855)
0810 11:38:10.421764 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:10.421780 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:10.421953 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:10.422307 (+ 354us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:10.422312 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:10.422321 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
0810 11:38:10.422652 (+ 331us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:10.422660 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.423761 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.423766 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:10.425236 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:10.425270 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.427725 (+ 2455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.427735 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:10.428033 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.428091 (+ 58us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0810 11:38:10.428628 (+ 537us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0810 11:38:10.428639 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 11:38:10.428717 (+ 78us) client_negotiation.cc:240] Negotiation successful
0810 11:38:10.428893 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1378,"thread_start_us":1215,"threads_started":1}
I20260810 11:38:10.440056 15402 raft_consensus.cc:1261] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:10.440200 15594 raft_consensus.cc:1261] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Refusing update from remote peer fc781f0fdf8144e0b3b7656f890bea23: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:10.440817 15714 consensus_queue.cc:1035] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4ea9d9fbfe18400c80b6d2d42a2121c7" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44747 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:10.441210 15809 consensus_queue.cc:1035] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [LEADER]: Connected to new peer: Peer: permanent_uuid: "01c16e176f13491bb3ca8c4518aaa24b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 45557 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:10.515086 14972 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:10.515388 14972 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52340:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:10.518625 14972 catalog_manager.cc:5782] T 00000000000000000000000000000000 P eed9d9b83f7a471596f17bb8d7573f0f: Sending DeleteTablet for 3 replicas of tablet debb8b729ef5432aa73e235d3b8a33ab
I20260810 11:38:10.519475 15171 tablet_service.cc:1514] Processing DeleteTablet for tablet debb8b729ef5432aa73e235d3b8a33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:51030
I20260810 11:38:10.519701 15568 tablet_service.cc:1514] Processing DeleteTablet for tablet debb8b729ef5432aa73e235d3b8a33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36330
I20260810 11:38:10.520004 15820 tablet_replica.cc:330] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: stopping tablet replica
I20260810 11:38:10.520000 15817 tablet_replica.cc:330] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: stopping tablet replica
I20260810 11:38:10.520145 15820 raft_consensus.cc:2229] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:10.520219 15817 raft_consensus.cc:2229] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:10.520313 15820 raft_consensus.cc:2258] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.520635 15817 raft_consensus.cc:2258] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.521271 15369 tablet_service.cc:1514] Processing DeleteTablet for tablet debb8b729ef5432aa73e235d3b8a33ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:10 UTC) from {username='oryx', principal='oryx/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42262
I20260810 11:38:10.521726 15820 ts_tablet_manager.cc:1905] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.521843 15819 tablet_replica.cc:330] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: stopping tablet replica
I20260810 11:38:10.522051 15819 raft_consensus.cc:2229] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:10.522287 15819 raft_consensus.cc:2258] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:10.523945 15817 ts_tablet_manager.cc:1905] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.524853 15819 ts_tablet_manager.cc:1905] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:10.531543 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 15081
I20260810 11:38:10.534721 15820 ts_tablet_manager.cc:1918] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:10.534890 15820 log.cc:1196] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/debb8b729ef5432aa73e235d3b8a33ab
I20260810 11:38:10.535355 15820 ts_tablet_manager.cc:1939] T debb8b729ef5432aa73e235d3b8a33ab P 01c16e176f13491bb3ca8c4518aaa24b: Deleting consensus metadata
I20260810 11:38:10.535926 15817 ts_tablet_manager.cc:1918] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:10.536108 15817 log.cc:1196] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/debb8b729ef5432aa73e235d3b8a33ab
I20260810 11:38:10.536545 15817 ts_tablet_manager.cc:1939] T debb8b729ef5432aa73e235d3b8a33ab P fc781f0fdf8144e0b3b7656f890bea23: Deleting consensus metadata
I20260810 11:38:10.537657 14953 catalog_manager.cc:4841] TS fc781f0fdf8144e0b3b7656f890bea23 (127.3.240.193:42427): tablet debb8b729ef5432aa73e235d3b8a33ab (table test-table [id=f511b43085014df4a610e7b2c1ea8a10]) successfully deleted
I20260810 11:38:10.538460 14950 catalog_manager.cc:4841] TS 01c16e176f13491bb3ca8c4518aaa24b (127.3.240.195:45557): tablet debb8b729ef5432aa73e235d3b8a33ab (table test-table [id=f511b43085014df4a610e7b2c1ea8a10]) successfully deleted
I20260810 11:38:10.539630 15819 ts_tablet_manager.cc:1918] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260810 11:38:10.539917 15819 log.cc:1196] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/debb8b729ef5432aa73e235d3b8a33ab
I20260810 11:38:10.540400 15819 ts_tablet_manager.cc:1939] T debb8b729ef5432aa73e235d3b8a33ab P 4ea9d9fbfe18400c80b6d2d42a2121c7: Deleting consensus metadata
I20260810 11:38:10.541692 14955 catalog_manager.cc:4841] TS 4ea9d9fbfe18400c80b6d2d42a2121c7 (127.3.240.194:44747): tablet debb8b729ef5432aa73e235d3b8a33ab (table test-table [id=f511b43085014df4a610e7b2c1ea8a10]) successfully deleted
Aug 10 11:38:10 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.193@KRBTEST.COM
I20260810 11:38:10.561590 15785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.533255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51054 (local address 127.3.240.193:42427)
0810 11:38:10.533463 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:10.533469 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:10.541930 (+ 8461us) server_negotiation.cc:407] Connection header received
0810 11:38:10.542430 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:10.542435 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:10.542509 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:10.542618 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:10.544614 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.545709 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.547653 (+ 1944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.548186 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.548311 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.554243 (+ 5932us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:10.554272 (+ 29us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:10.554317 (+ 45us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:10.554472 (+ 155us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:10.558225 (+ 3753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.559024 (+ 799us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.559032 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.559034 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.559216 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.559844 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.559849 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.559851 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.560232 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:10.560584 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:10.560904 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 11:38:10.561160 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20260810 11:38:10.562750 15266 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:10.734921 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 15081
I20260810 11:38:10.775979 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 15276
Aug 10 11:38:10 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.194@KRBTEST.COM
I20260810 11:38:10.801169 15873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.776747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42280 (local address 127.3.240.194:44747)
0810 11:38:10.783714 (+ 6967us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:10.783720 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:10.783742 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:38:10.783801 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:10.783805 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:10.783891 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:10.784056 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:10.785379 (+ 1323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.786188 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.787595 (+ 1407us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.788026 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.788180 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.793844 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:10.793864 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:10.793883 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:10.793931 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:10.797489 (+ 3558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.798325 (+ 836us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.798333 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.798335 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.798512 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.799138 (+ 626us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.799190 (+ 52us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.799226 (+ 36us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.799730 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:10.800094 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:10.800416 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 11:38:10.800645 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6673,"thread_start_us":6546,"threads_started":1}
I20260810 11:38:10.802536 15462 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:10.939772 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 15276
I20260810 11:38:10.976400 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 15475
Aug 10 11:38:10 dist-test-slave-drl0 krb5kdc[14907](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361889, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.3.240.195@KRBTEST.COM
I20260810 11:38:10.995930 15875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:10.977333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36336 (local address 127.3.240.195:45557)
0810 11:38:10.977977 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:10.977982 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:10.978000 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:38:10.978058 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:10.978061 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:10.978128 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:10.978239 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:10.979768 (+ 1529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.980488 (+ 720us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.982343 (+ 1855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:10.982870 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:10.983298 (+ 428us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:10.989017 (+ 5719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:10.989039 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:10.989064 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:10.989116 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:10.992942 (+ 3826us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.993697 (+ 755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.993706 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.993708 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.993855 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:10.994297 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:10.994301 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:10.994303 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:10.994669 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:10.995107 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:10.995406 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 11:38:10.995603 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":383,"threads_started":1}
I20260810 11:38:10.996752 15661 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:11.128950 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 15475
I20260810 11:38:11.162869 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 14922
I20260810 11:38:11.164508 15034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:11.288295 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 14922
2026-08-10T11:38:11Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (8076 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:38:11 dist-test-slave-drl0 krb5kdc[15884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:38:11 dist-test-slave-drl0 krb5kdc[15884](info): set up 2 sockets
Aug 10 11:38:11 dist-test-slave-drl0 krb5kdc[15884](info): commencing operation
krb5kdc: starting...
W20260810 11:38:12.700333 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.351s user 0.004s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:38:12 dist-test-slave-drl0 krb5kdc[15884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361892, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:38:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:38:13.056622 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36061
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40307
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36061
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:13.444170 15903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:13.444491 15903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:13.444650 15903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:13.452543 15903 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:13.452657 15903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:13.452732 15903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:13.452816 15903 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:13.452886 15903 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:13.463587 15903 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40307
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36061
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36061
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:13.465184 15903 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:13.467558 15903 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:13.483140 15911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:13.483146 15912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:13.483045 15917 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:13.484437 15903 server_base.cc:1034] running on GCE node
I20260810 11:38:13.485543 15903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:13.487663 15903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:13.489125 15903 hybrid_clock.cc:648] HybridClock initialized: now 1786361893489064 us; error 54 us; skew 500 ppm
I20260810 11:38:13.489615 15903 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-10T11:38:13Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2264 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:38:13 dist-test-slave-drl0 krb5kdc[15926](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:38:13 dist-test-slave-drl0 krb5kdc[15926](info): set up 2 sockets
Aug 10 11:38:13 dist-test-slave-drl0 krb5kdc[15926](info): commencing operation
krb5kdc: starting...
W20260810 11:38:14.924787 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.311s user 0.010s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:38:15 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:38:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:38:15.259480 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39665
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:15.641808 15945 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:15.642110 15945 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:15.642202 15945 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:15.651077 15945 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:15.651197 15945 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:15.651245 15945 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:15.651304 15945 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:15.651343 15945 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:15.662150 15945 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:15.663599 15945 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:15.665134 15945 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:15.677767 15950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:15.677837 15951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:15.677826 15953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:15.678637 15945 server_base.cc:1034] running on GCE node
I20260810 11:38:15.679622 15945 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:15.681144 15945 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:15.682425 15945 hybrid_clock.cc:648] HybridClock initialized: now 1786361895682323 us; error 86 us; skew 500 ppm
I20260810 11:38:15.682825 15945 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:15 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:15.692209 15945 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:38:15.693876 15945 webserver.cc:460] Webserver started at http://127.3.240.254:36763/ using document root <none> and password file <none>
I20260810 11:38:15.694620 15945 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:15.694766 15945 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:15.695170 15945 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:15.698220 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "16ec63ea07564201bb16218fe1dbcb0c"
format_stamp: "Formatted at 2026-08-10 11:38:15 on dist-test-slave-drl0"
server_key: "2d9b1cd59142608685468685d6dcf31a"
server_key_iv: "8778bde2aa9e06b01a15726ded777d78"
server_key_version: "encryptionkey@0"
I20260810 11:38:15.699054 15945 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "16ec63ea07564201bb16218fe1dbcb0c"
format_stamp: "Formatted at 2026-08-10 11:38:15 on dist-test-slave-drl0"
server_key: "2d9b1cd59142608685468685d6dcf31a"
server_key_iv: "8778bde2aa9e06b01a15726ded777d78"
server_key_version: "encryptionkey@0"
I20260810 11:38:15.705226 15945 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:38:15.708904 15965 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:15.710574 15945 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:38:15.710809 15945 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "16ec63ea07564201bb16218fe1dbcb0c"
format_stamp: "Formatted at 2026-08-10 11:38:15 on dist-test-slave-drl0"
server_key: "2d9b1cd59142608685468685d6dcf31a"
server_key_iv: "8778bde2aa9e06b01a15726ded777d78"
server_key_version: "encryptionkey@0"
I20260810 11:38:15.711105 15945 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:15.754339 15945 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:15.759697 15945 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:15.760162 15945 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:15.783023 15945 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39665
I20260810 11:38:15.783020 16033 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39665 every 8 connection(s)
I20260810 11:38:15.784507 15945 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:15.787510 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 15945
I20260810 11:38:15.787739 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:38:15.788576 4035 external_mini_cluster.cc:1428] Setting key 07b136ffbb684aacaf6cacaffcf6d930
I20260810 11:38:15.789600 16034 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 11:38:15 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:15.805539 16034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Bootstrap starting.
I20260810 11:38:15.809266 16034 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:15.810412 16034 log.cc:824] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:15.814533 16034 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: No bootstrap required, opened a new log
I20260810 11:38:15.815933 16041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:15.791275 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41312 (local address 127.3.240.254:39665)
0810 11:38:15.792200 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:15.792212 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:15.792279 (+ 67us) server_negotiation.cc:407] Connection header received
0810 11:38:15.794766 (+ 2487us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:15.794789 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:15.795271 (+ 482us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:15.795689 (+ 418us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:15.797282 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:15.798634 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:15.800099 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:15.800787 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:15.801021 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:15.806133 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:15.806151 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:15.806174 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:15.806231 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:15.810235 (+ 4004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:15.811679 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:15.811688 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:15.811696 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:15.811939 (+ 243us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:15.812426 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:15.812430 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:15.812432 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:15.813212 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:15.813602 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:15.814148 (+ 546us) server_negotiation.cc:299] Negotiation successful
0810 11:38:15.814687 (+ 539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":383,"threads_started":1}
I20260810 11:38:15.824738 16034 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:15.825122 16034 raft_consensus.cc:374] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:15.825239 16034 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ec63ea07564201bb16218fe1dbcb0c, State: Initialized, Role: FOLLOWER
I20260810 11:38:15.825932 16034 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:15.826170 16034 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:15.826323 16034 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:15.826507 16034 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:15.828845 16034 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:15.829372 16034 leader_election.cc:302] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ec63ea07564201bb16218fe1dbcb0c; no voters:
I20260810 11:38:15.830184 16034 leader_election.cc:288] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:38:15.830354 16043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:15.832482 16034 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:15.835423 16043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 LEADER]: Becoming Leader. State: Replica: 16ec63ea07564201bb16218fe1dbcb0c, State: Running, Role: LEADER
I20260810 11:38:15.836076 16043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:15.839362 16044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "16ec63ea07564201bb16218fe1dbcb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } } }
I20260810 11:38:15.839754 16044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:15.840284 16045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ec63ea07564201bb16218fe1dbcb0c. Latest consensus state: current_term: 1 leader_uuid: "16ec63ea07564201bb16218fe1dbcb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } } }
I20260810 11:38:15.840581 16045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:15.841511 16068 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:15.847554 16068 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:15.856786 16068 catalog_manager.cc:1347] Generated new cluster ID: 180a3d1fef1140f585f6ccba722978c1
I20260810 11:38:15.856915 16068 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:15.880059 16068 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:38:15.881592 16068 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:15.891592 16068 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Generated new TSK 0
I20260810 11:38:15.892508 16068 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:38:15.976851 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:16.350330 16077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:16.350605 16077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:16.350715 16077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:16.358104 16077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:16.358260 16077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:16.358373 16077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:16.368621 16077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:16.370044 16077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:16.371443 16077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:16.387400 16086 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:16.387549 16085 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:16.387936 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:16.389060 16077 server_base.cc:1034] running on GCE node
I20260810 11:38:16.389825 16077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:16.391289 16077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:16.392561 16077 hybrid_clock.cc:648] HybridClock initialized: now 1786361896392493 us; error 52 us; skew 500 ppm
I20260810 11:38:16.392979 16077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:16 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361896, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:16.398293 16077 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:38:16.400689 16077 webserver.cc:460] Webserver started at http://127.3.240.193:38557/ using document root <none> and password file <none>
I20260810 11:38:16.401743 16077 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:16.401901 16077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:16.402338 16077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:16.406773 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "6a24f1cf6f1240fd9b488060f5c152cb"
format_stamp: "Formatted at 2026-08-10 11:38:16 on dist-test-slave-drl0"
server_key: "fafa651436246f8d750cd0a6e7d86dd6"
server_key_iv: "70a72293ca5a1466b54518e95974aff2"
server_key_version: "encryptionkey@0"
I20260810 11:38:16.407998 16077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "6a24f1cf6f1240fd9b488060f5c152cb"
format_stamp: "Formatted at 2026-08-10 11:38:16 on dist-test-slave-drl0"
server_key: "fafa651436246f8d750cd0a6e7d86dd6"
server_key_iv: "70a72293ca5a1466b54518e95974aff2"
server_key_version: "encryptionkey@0"
I20260810 11:38:16.415481 16077 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 11:38:16.419680 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:16.421690 16077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:38:16.421901 16077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "6a24f1cf6f1240fd9b488060f5c152cb"
format_stamp: "Formatted at 2026-08-10 11:38:16 on dist-test-slave-drl0"
server_key: "fafa651436246f8d750cd0a6e7d86dd6"
server_key_iv: "70a72293ca5a1466b54518e95974aff2"
server_key_version: "encryptionkey@0"
I20260810 11:38:16.422151 16077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:16.446209 16077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:16.453437 16077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:16.453902 16077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:16.455173 16077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:16.457494 16077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:16.457615 16077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:16.457821 16077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:16.457897 16077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:16.500939 16077 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43585
I20260810 11:38:16.501155 16294 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43585 every 8 connection(s)
I20260810 11:38:16.502552 16077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:16.510167 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 16077
I20260810 11:38:16.510387 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:38:16.511264 4035 external_mini_cluster.cc:1428] Setting key d0d04f3e1c0e45a75f26fa8ccdf247fc
Aug 10 11:38:16 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361896, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:16.525070 16298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:16.505371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47861 (local address 127.3.240.254:39665)
0810 11:38:16.505861 (+ 490us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:16.505868 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:16.507293 (+ 1425us) server_negotiation.cc:407] Connection header received
0810 11:38:16.509057 (+ 1764us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:16.509062 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:16.509132 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:16.509241 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:16.510832 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:16.511899 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:16.513478 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:16.513861 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:16.514033 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:16.518624 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:16.518635 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:16.518638 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:16.518670 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:16.521405 (+ 2735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:16.522487 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:16.522492 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:16.522494 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:16.522617 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:16.523261 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:16.523265 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:16.523268 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:16.523543 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:16.523682 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:16.524616 (+ 934us) server_negotiation.cc:299] Negotiation successful
0810 11:38:16.524793 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":209,"threads_started":1}
I20260810 11:38:16.526389 16299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:16.506012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:47861)
0810 11:38:16.507116 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:16.507175 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:16.508784 (+ 1609us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:16.509442 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:16.509449 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:16.510004 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:16.510562 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:16.510579 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:16.512092 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:16.512097 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:16.513305 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:16.513317 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:16.514124 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:16.514134 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:16.514634 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:16.515621 (+ 987us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:16.515645 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:16.518378 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:16.521552 (+ 3174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:16.521638 (+ 86us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:16.521668 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:16.522321 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:16.522801 (+ 480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:16.522809 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:16.522815 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:16.523092 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:16.523685 (+ 593us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:16.523695 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:16.524165 (+ 470us) client_negotiation.cc:769] Sending connection context
0810 11:38:16.524485 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 11:38:16.525046 (+ 561us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260810 11:38:16.528081 16295 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39665
I20260810 11:38:16.528458 16295 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:16.529472 16295 heartbeater.cc:511] Master 127.3.240.254:39665 requested a full tablet report, sending...
I20260810 11:38:16.531956 15988 ts_manager.cc:194] Registered new tserver with Master: 6a24f1cf6f1240fd9b488060f5c152cb (127.3.240.193:43585)
I20260810 11:38:16.533628 15988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:47861
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:38:16.579610 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:16.976238 16303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:16.976601 16303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:16.976783 16303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:16.984973 16303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:16.985133 16303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:16.985308 16303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:16.998543 16303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:17.000061 16303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:17.001507 16303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:17.015763 16317 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:17.015791 16314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:17.015955 16320 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:17.017100 16303 server_base.cc:1034] running on GCE node
I20260810 11:38:17.017776 16303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:17.019429 16303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:17.020746 16303 hybrid_clock.cc:648] HybridClock initialized: now 1786361897020681 us; error 58 us; skew 500 ppm
I20260810 11:38:17.021130 16303 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:17 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:17.024547 16303 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:38:17.026322 16303 webserver.cc:460] Webserver started at http://127.3.240.194:46299/ using document root <none> and password file <none>
I20260810 11:38:17.027225 16303 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:17.027379 16303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:17.027709 16303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:17.030726 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "01ec73b87a674074a901ce1d9ee78767"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "af83ebffec2bdcde049f2bfcedbc79c2"
server_key_iv: "8124662d61e2548c71bb996425799eb6"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.031601 16303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "01ec73b87a674074a901ce1d9ee78767"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "af83ebffec2bdcde049f2bfcedbc79c2"
server_key_iv: "8124662d61e2548c71bb996425799eb6"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.037833 16303 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:38:17.042502 16333 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.044198 16303 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:38:17.044497 16303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "01ec73b87a674074a901ce1d9ee78767"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "af83ebffec2bdcde049f2bfcedbc79c2"
server_key_iv: "8124662d61e2548c71bb996425799eb6"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.044727 16303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:17.067910 16303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:17.072882 16303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:17.073338 16303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:17.074827 16303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:17.076762 16303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:17.076889 16303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.077061 16303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:17.077286 16303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.115067 16303 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:37653
I20260810 11:38:17.115197 16498 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:37653 every 8 connection(s)
I20260810 11:38:17.116671 16303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:17.125847 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 16303
I20260810 11:38:17.126044 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:38:17.126500 4035 external_mini_cluster.cc:1428] Setting key 85a9c1d5c601f6f42eb501d6c79653e8
Aug 10 11:38:17 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:17.141798 16501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:17.120190 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:41973 (local address 127.3.240.254:39665)
0810 11:38:17.120691 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:17.120696 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:17.121824 (+ 1128us) server_negotiation.cc:407] Connection header received
0810 11:38:17.123912 (+ 2088us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:17.123917 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:17.123986 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:17.124094 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:17.126355 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.127206 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.128895 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.129565 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.129675 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:17.135542 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:17.135547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:17.135550 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:17.135583 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:17.138584 (+ 3001us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.139416 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.139427 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.139431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.139551 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.140003 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.140008 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.140012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.140292 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:17.140450 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:17.141175 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 11:38:17.141409 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":279,"threads_started":1}
I20260810 11:38:17.142819 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:17.120803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:41973)
0810 11:38:17.121658 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:17.121700 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:17.123629 (+ 1929us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:17.124542 (+ 913us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:17.124554 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:17.125431 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:17.126144 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:17.126178 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.127361 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.127367 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:17.128708 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:17.128725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.129820 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.129829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:17.130162 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:17.131019 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:17.131056 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:17.135274 (+ 4218us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:17.138738 (+ 3464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:17.138749 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:17.138766 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:17.139284 (+ 518us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:17.139666 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:17.139671 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:17.139675 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:17.139876 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:17.140427 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:17.140435 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:17.140792 (+ 357us) client_negotiation.cc:769] Sending connection context
0810 11:38:17.141080 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 11:38:17.141630 (+ 550us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":338,"threads_started":1}
I20260810 11:38:17.144618 16499 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39665
I20260810 11:38:17.144963 16499 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:17.145860 16499 heartbeater.cc:511] Master 127.3.240.254:39665 requested a full tablet report, sending...
I20260810 11:38:17.148125 15988 ts_manager.cc:194] Registered new tserver with Master: 01ec73b87a674074a901ce1d9ee78767 (127.3.240.194:37653)
I20260810 11:38:17.149016 15988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:41973
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:38:17.202557 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:38:17.537511 16295 heartbeater.cc:503] Master 127.3.240.254:39665 was elected leader, sending a full tablet report...
W20260810 11:38:17.583073 16516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:17.583369 16516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:17.583528 16516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:17.590832 16516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:17.590984 16516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:17.591208 16516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:17.601109 16516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:17.602877 16516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:17.604313 16516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:17.618960 16522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:17.618848 16527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:17.619055 16524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:17.625324 16516 server_base.cc:1034] running on GCE node
I20260810 11:38:17.626171 16516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:17.627394 16516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:17.628664 16516 hybrid_clock.cc:648] HybridClock initialized: now 1786361897628609 us; error 46 us; skew 500 ppm
I20260810 11:38:17.629048 16516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:17 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:17.633430 16516 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:38:17.635720 16516 webserver.cc:460] Webserver started at http://127.3.240.195:45897/ using document root <none> and password file <none>
I20260810 11:38:17.636687 16516 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:17.636865 16516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:17.637374 16516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:17.640611 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "5aee1a380e844f048054a3855cca1b98"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "9bb612263457b0a0108cbc0678a71d64"
server_key_iv: "a87fb48e947c9b943ff657eea6002b88"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.641461 16516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "5aee1a380e844f048054a3855cca1b98"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "9bb612263457b0a0108cbc0678a71d64"
server_key_iv: "a87fb48e947c9b943ff657eea6002b88"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.647768 16516 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 11:38:17.653733 16536 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.655978 16516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:17.656180 16516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "5aee1a380e844f048054a3855cca1b98"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "9bb612263457b0a0108cbc0678a71d64"
server_key_iv: "a87fb48e947c9b943ff657eea6002b88"
server_key_version: "encryptionkey@0"
I20260810 11:38:17.656531 16516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:17.679078 16516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:17.685501 16516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:17.685835 16516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:17.687208 16516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:17.688807 16516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:17.688930 16516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.689105 16516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:17.689178 16516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:17.730173 16516 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33761
I20260810 11:38:17.730260 16710 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33761 every 8 connection(s)
I20260810 11:38:17.731528 16516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:17.737191 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 16516
I20260810 11:38:17.737408 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:38:17.737993 4035 external_mini_cluster.cc:1428] Setting key b19c380c1e7d9a8a3aa6962c528d374e
Aug 10 11:38:17 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361897, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:17.756577 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:17.734746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55303 (local address 127.3.240.254:39665)
0810 11:38:17.735673 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:17.735678 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:17.736639 (+ 961us) server_negotiation.cc:407] Connection header received
0810 11:38:17.738743 (+ 2104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:17.738750 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:17.738888 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:17.739041 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:17.741064 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.742000 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.744067 (+ 2067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.744493 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.744636 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:17.749216 (+ 4580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:17.749223 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:17.749227 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:17.749271 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:17.752544 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.753587 (+ 1043us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.753593 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.753598 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.753750 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.754380 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.754386 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.754389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.754906 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:17.755234 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:17.755959 (+ 725us) server_negotiation.cc:299] Negotiation successful
0810 11:38:17.756209 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":754,"thread_start_us":563,"threads_started":1}
I20260810 11:38:17.757443 16715 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:17.735423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:55303)
0810 11:38:17.736325 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:17.736374 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:17.738438 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:17.739428 (+ 990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:17.739442 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:17.740109 (+ 667us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:17.740839 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:17.740870 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.742172 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.742181 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:17.743898 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:17.743911 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.744733 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.744741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:17.744967 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:17.745701 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:17.745742 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:17.748831 (+ 3089us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:17.752726 (+ 3895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:17.752737 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:17.752760 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:17.753411 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:17.753884 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:17.753891 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:17.753895 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:17.754149 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:17.755101 (+ 952us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:17.755108 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:17.755534 (+ 426us) client_negotiation.cc:769] Sending connection context
0810 11:38:17.755801 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 11:38:17.756218 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":379,"threads_started":1}
I20260810 11:38:17.759052 16711 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39665
I20260810 11:38:17.759342 16711 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:17.760075 16711 heartbeater.cc:511] Master 127.3.240.254:39665 requested a full tablet report, sending...
I20260810 11:38:17.761739 15988 ts_manager.cc:194] Registered new tserver with Master: 5aee1a380e844f048054a3855cca1b98 (127.3.240.195:33761)
I20260810 11:38:17.762701 15988 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:55303
I20260810 11:38:17.768505 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
Entry for principal oryx/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab.
WARNING: no policy specified for oryx/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
Entry for principal oryx/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab.
WARNING: no policy specified for oryx/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
Entry for principal oryx/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab.
I20260810 11:38:17.976743 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16077
Aug 10 11:38:17 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:38:17.997871 16725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:17.978239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47316 (local address 127.3.240.193:43585)
0810 11:38:17.979265 (+ 1026us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:17.979274 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:17.979312 (+ 38us) server_negotiation.cc:407] Connection header received
0810 11:38:17.979441 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:17.979449 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:17.979747 (+ 298us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:17.979913 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:17.981713 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.983262 (+ 1549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.984948 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:17.985577 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:17.985708 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:17.990674 (+ 4966us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:17.990689 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:17.990717 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:17.990761 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:17.993938 (+ 3177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.994782 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.994793 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.994796 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.995181 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:17.995762 (+ 581us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:17.995768 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:17.995770 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:17.996200 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:17.996473 (+ 273us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:17.996958 (+ 485us) server_negotiation.cc:299] Negotiation successful
0810 11:38:17.997480 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":561,"threads_started":1}
I20260810 11:38:17.999686 16289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:18.091554 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 16077
I20260810 11:38:18.118637 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16303
Aug 10 11:38:18 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:38:18.137926 16730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:18.119977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43358 (local address 127.3.240.194:37653)
0810 11:38:18.120809 (+ 832us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:18.120822 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:18.120865 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:38:18.120984 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:18.120994 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:18.121437 (+ 443us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:18.121640 (+ 203us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:18.122960 (+ 1320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:18.124197 (+ 1237us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:18.125815 (+ 1618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:18.126426 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:18.126544 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:18.131824 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:18.131843 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:18.131860 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:18.131907 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:18.134871 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:18.135669 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:18.135676 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:18.135678 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:18.135819 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:18.136204 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:18.136209 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:18.136212 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:18.136586 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:18.136950 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:18.137242 (+ 292us) server_negotiation.cc:299] Negotiation successful
0810 11:38:18.137528 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":374,"threads_started":1}
I20260810 11:38:18.139463 16493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:18.226405 16499 heartbeater.cc:503] Master 127.3.240.254:39665 was elected leader, sending a full tablet report...
I20260810 11:38:18.227526 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 16303
W20260810 11:38:18.255092 15966 connection.cc:535] server connection from 127.3.240.194:41973 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 11:38:18.255626 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16516
Aug 10 11:38:18 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361895, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:38:18.274130 16735 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:18.256929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39902 (local address 127.3.240.195:33761)
0810 11:38:18.257573 (+ 644us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:18.257581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:18.257614 (+ 33us) server_negotiation.cc:407] Connection header received
0810 11:38:18.257680 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:18.257686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:18.257910 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:18.258055 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:18.259583 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:18.260578 (+ 995us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:18.262007 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:18.262637 (+ 630us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:18.263083 (+ 446us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:18.267748 (+ 4665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:18.267760 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:18.267774 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:18.267816 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:18.271011 (+ 3195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:18.271928 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:18.271936 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:18.271938 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:18.272079 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:18.272575 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:18.272579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:18.272581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:18.272977 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:18.273220 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:18.273616 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 11:38:18.273827 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":327,"threads_started":1}
I20260810 11:38:18.275442 16705 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:18.365917 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 16516
I20260810 11:38:18.393088 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 15945
I20260810 11:38:18.394532 16029 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:18.510425 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 15945
I20260810 11:38:18.541608 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39665
--webserver_interface=127.3.240.254
--webserver_port=36763
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39665
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:18.933954 16739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:18.934386 16739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:18.934572 16739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:18.944748 16739 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:18.944972 16739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:18.945089 16739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:18.945204 16739 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:18.945299 16739 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:18.957284 16739 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39665
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=36763
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:18.958770 16739 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:18.960278 16739 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:18.974160 16747 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:18.974274 16746 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:18.974249 16751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:18.975306 16739 server_base.cc:1034] running on GCE node
I20260810 11:38:18.976490 16739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:18.978444 16739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:18.979863 16739 hybrid_clock.cc:648] HybridClock initialized: now 1786361898979821 us; error 66 us; skew 500 ppm
I20260810 11:38:18.980326 16739 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:18 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361898, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:18.983851 16739 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:38:18.985541 16739 webserver.cc:460] Webserver started at http://127.3.240.254:36763/ using document root <none> and password file <none>
I20260810 11:38:18.986445 16739 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:18.986642 16739 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:18.993748 16739 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.002s
I20260810 11:38:18.997481 16762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:18.999305 16739 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:18.999524 16739 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "16ec63ea07564201bb16218fe1dbcb0c"
format_stamp: "Formatted at 2026-08-10 11:38:15 on dist-test-slave-drl0"
server_key: "2d9b1cd59142608685468685d6dcf31a"
server_key_iv: "8778bde2aa9e06b01a15726ded777d78"
server_key_version: "encryptionkey@0"
I20260810 11:38:19.000559 16739 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:19.023666 16739 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:19.028352 16739 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:19.028649 16739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:19.049568 16739 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39665
I20260810 11:38:19.049597 16834 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39665 every 8 connection(s)
I20260810 11:38:19.052691 16739 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:19.060353 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 16739
I20260810 11:38:19.060557 16835 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:38:19.061439 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:43585
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:39665
--webserver_port=38557
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:38:19.066031 16835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Bootstrap starting.
I20260810 11:38:19.090430 16835 log.cc:824] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:19.100087 16835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:38:19.100656 16835 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Bootstrap complete.
I20260810 11:38:19.111184 16835 raft_consensus.cc:348] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:19.112027 16835 raft_consensus.cc:729] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 16ec63ea07564201bb16218fe1dbcb0c, State: Initialized, Role: FOLLOWER
I20260810 11:38:19.112785 16835 consensus_queue.cc:260] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:19.113037 16835 raft_consensus.cc:388] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:19.113191 16835 raft_consensus.cc:482] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:19.113410 16835 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:38:19.117280 16835 raft_consensus.cc:504] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:19.117789 16835 leader_election.cc:302] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 16ec63ea07564201bb16218fe1dbcb0c; no voters:
I20260810 11:38:19.118450 16835 leader_election.cc:288] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 11:38:19.118593 16840 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 2 FOLLOWER]: Leader election won for term 2
I20260810 11:38:19.120976 16835 sys_catalog.cc:564] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:19.123449 16840 raft_consensus.cc:686] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [term 2 LEADER]: Becoming Leader. State: Replica: 16ec63ea07564201bb16218fe1dbcb0c, State: Running, Role: LEADER
I20260810 11:38:19.124207 16840 consensus_queue.cc:237] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } }
I20260810 11:38:19.129201 16844 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 16ec63ea07564201bb16218fe1dbcb0c. Latest consensus state: current_term: 2 leader_uuid: "16ec63ea07564201bb16218fe1dbcb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } } }
I20260810 11:38:19.129303 16841 sys_catalog.cc:455] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "16ec63ea07564201bb16218fe1dbcb0c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "16ec63ea07564201bb16218fe1dbcb0c" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39665 } } }
I20260810 11:38:19.129755 16844 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:19.129761 16841 sys_catalog.cc:458] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:19.143576 16852 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:19.150401 16852 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:19.152132 16852 catalog_manager.cc:1259] Loaded cluster ID: 180a3d1fef1140f585f6ccba722978c1
I20260810 11:38:19.152318 16852 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:19.156563 16852 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:19.158257 16852 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 16ec63ea07564201bb16218fe1dbcb0c: Loaded TSK: 0
I20260810 11:38:19.160372 16852 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 11:38:19.468554 16837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:19.468886 16837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:19.469058 16837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:19.476562 16837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:19.476713 16837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:19.476886 16837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:19.486915 16837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.193.keytab
--principal=oryx/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:43585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=38557
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:19.489220 16837 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:19.490705 16837 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:19.507333 16873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:19.507149 16876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:19.509351 16881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:19.509742 16837 server_base.cc:1034] running on GCE node
I20260810 11:38:19.510453 16837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:19.511765 16837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:19.513026 16837 hybrid_clock.cc:648] HybridClock initialized: now 1786361899512974 us; error 37 us; skew 500 ppm
I20260810 11:38:19.513414 16837 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:19 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361899, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:19.517071 16837 init.cc:377] Logged in from keytab as oryx/127.3.240.193@KRBTEST.COM (short username oryx)
I20260810 11:38:19.518710 16837 webserver.cc:460] Webserver started at http://127.3.240.193:38557/ using document root <none> and password file <none>
I20260810 11:38:19.519526 16837 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:19.519690 16837 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:19.526866 16837 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.001s
I20260810 11:38:19.530508 16889 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:19.532095 16837 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:19.532297 16837 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "6a24f1cf6f1240fd9b488060f5c152cb"
format_stamp: "Formatted at 2026-08-10 11:38:16 on dist-test-slave-drl0"
server_key: "fafa651436246f8d750cd0a6e7d86dd6"
server_key_iv: "70a72293ca5a1466b54518e95974aff2"
server_key_version: "encryptionkey@0"
I20260810 11:38:19.533311 16837 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:19.547463 16837 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:19.552446 16837 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:19.552757 16837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:19.553789 16837 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:19.555295 16837 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:19.555431 16837 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:19.555572 16837 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:19.555665 16837 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:19.596208 16837 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:43585
I20260810 11:38:19.596297 17055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:43585 every 8 connection(s)
I20260810 11:38:19.597730 16837 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:19.599526 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 16837
I20260810 11:38:19.600147 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:37653
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:39665
--webserver_port=46299
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:38:19 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:19 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 11:38:19.973008 17062 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:19.973366 17062 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:19.973537 17062 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:19.981242 17062 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:19.981400 17062 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:19.981580 17062 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:19.991847 17062 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.194.keytab
--principal=oryx/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:37653
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=46299
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:19.993268 17062 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:19.994616 17062 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:20.013449 17074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:20.013778 17073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:20.013468 17077 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:20.015211 17062 server_base.cc:1034] running on GCE node
I20260810 11:38:20.016319 17062 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:20.018028 17062 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:20.019419 17062 hybrid_clock.cc:648] HybridClock initialized: now 1786361900019331 us; error 54 us; skew 500 ppm
I20260810 11:38:20.019990 17062 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:20 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361900, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:20.026468 17062 init.cc:377] Logged in from keytab as oryx/127.3.240.194@KRBTEST.COM (short username oryx)
I20260810 11:38:20.029016 17062 webserver.cc:460] Webserver started at http://127.3.240.194:46299/ using document root <none> and password file <none>
I20260810 11:38:20.030354 17062 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:20.030573 17062 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:20.041640 17062 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.003s
I20260810 11:38:20.046754 17086 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:20.049906 17062 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260810 11:38:20.050199 17062 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "01ec73b87a674074a901ce1d9ee78767"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "af83ebffec2bdcde049f2bfcedbc79c2"
server_key_iv: "8124662d61e2548c71bb996425799eb6"
server_key_version: "encryptionkey@0"
I20260810 11:38:20.052428 17062 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:20.092051 17062 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:20.096781 17062 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:20.097074 17062 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:20.098205 17062 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:20.099812 17062 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:20.099943 17062 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:20.100080 17062 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:20.100170 17062 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:20.153781 17062 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:37653
I20260810 11:38:20.153841 17249 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:37653 every 8 connection(s)
I20260810 11:38:20.155381 17062 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:20.158515 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 17062
I20260810 11:38:20.159337 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:33761
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:39665
--webserver_port=45897
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42383
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:38:20 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:20 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
W20260810 11:38:20.878291 17255 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:20.878705 17255 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:20.878852 17255 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:20.891693 17255 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:20.891824 17255 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:20.892019 17255 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:20.903311 17255 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42383
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/krb5kdc/oryx_127.3.240.195.keytab
--principal=oryx/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:33761
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=45897
--tserver_master_addrs=127.3.240.254:39665
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:20.904814 17255 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:20.906329 17255 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:20.925048 17267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:20.928376 17266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:20.931105 17269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:20.932150 17255 server_base.cc:1034] running on GCE node
I20260810 11:38:20.933182 17255 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:20.937119 17255 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:20.940544 17255 hybrid_clock.cc:648] HybridClock initialized: now 1786361900940286 us; error 244 us; skew 500 ppm
I20260810 11:38:20.941145 17255 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:20 dist-test-slave-drl0 krb5kdc[15926](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361900, etypes {rep=17 tkt=17 ses=17}, oryx/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:20.948186 17255 init.cc:377] Logged in from keytab as oryx/127.3.240.195@KRBTEST.COM (short username oryx)
I20260810 11:38:20.950176 17255 webserver.cc:460] Webserver started at http://127.3.240.195:45897/ using document root <none> and password file <none>
I20260810 11:38:20.951037 17255 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:20.951208 17255 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:20.957538 17255 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:38:20.960485 17276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:20.962515 17255 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:20.963003 17255 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "5aee1a380e844f048054a3855cca1b98"
format_stamp: "Formatted at 2026-08-10 11:38:17 on dist-test-slave-drl0"
server_key: "9bb612263457b0a0108cbc0678a71d64"
server_key_iv: "a87fb48e947c9b943ff657eea6002b88"
server_key_version: "encryptionkey@0"
I20260810 11:38:20.964490 17255 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260810 11:38:20.984828 17061 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.373s user 0.009s sys 0.010s
I20260810 11:38:20.985630 17255 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
W20260810 11:38:20.988515 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:19.601171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35755 (local address 127.3.240.254:39665)
0810 11:38:19.603052 (+ 1881us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:19.603068 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:19.603121 (+ 53us) server_negotiation.cc:407] Connection header received
0810 11:38:19.604781 (+ 1660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:19.604793 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:19.605208 (+ 415us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:19.605586 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:19.607118 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:19.608487 (+ 1369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:19.610069 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:19.610695 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:19.610899 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:20.986972 (+1376073us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:35755: BlockingRecv error: recv got EOF from 127.3.240.193:35755 (error 108)
Metrics: {"server-negotiator.queue_time_us":1586,"thread_start_us":1267,"threads_started":1}
I20260810 11:38:20.988680 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:19.601233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:35755)
0810 11:38:19.602075 (+ 842us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:19.602129 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:19.604246 (+ 2117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:19.605705 (+ 1459us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:19.605716 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:19.606347 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:19.606862 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:19.606878 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:19.608673 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:19.608685 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:19.609916 (+ 1231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:19.609929 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:19.610977 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:19.610987 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:19.611259 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:19.611972 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:19.612009 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:20.987269 (+1375260us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":370,"threads_started":1}
W20260810 11:38:20.989977 17057 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:39665 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:39665: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
I20260810 11:38:20.994074 17255 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:20.994439 17255 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:20.996014 17255 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:20.997974 17255 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:20.998100 17255 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:20.998217 17255 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:20.998322 17255 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:38:20 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:21.007129 17061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:20.990820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:39579)
0810 11:38:20.991142 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:20.991162 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:20.991306 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:20.992013 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:20.992018 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:20.992523 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:20.992786 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:20.992802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.995530 (+ 2728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.995539 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:20.997181 (+ 1642us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:20.997205 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.998569 (+ 1364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.998579 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:20.998842 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:20.999653 (+ 811us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:20.999676 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:21.006591 (+ 6915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":196}
W20260810 11:38:21.007827 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:20.990848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:39579 (local address 127.3.240.254:39665)
0810 11:38:20.991125 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:20.991133 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:20.991224 (+ 91us) server_negotiation.cc:407] Connection header received
0810 11:38:20.991504 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:20.991513 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:20.991633 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:20.991808 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:20.992994 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.995205 (+ 2211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.997454 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.998131 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.998254 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.007219 (+ 8965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:39579: BlockingRecv error: recv got EOF from 127.3.240.193:39579 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:21.025619 17299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:21.008305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:39549)
0810 11:38:21.009137 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:21.009166 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:21.009435 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:21.009945 (+ 510us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:21.009952 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:21.010882 (+ 930us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:21.011341 (+ 459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.011360 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.013286 (+ 1926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.013295 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.015819 (+ 2524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.015845 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.017260 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.017272 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.017601 (+ 329us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.018636 (+ 1035us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:21.018673 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:21.025068 (+ 6395us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":630,"thread_start_us":791,"threads_started":1}
W20260810 11:38:21.025767 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:21.008894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:39549 (local address 127.3.240.254:39665)
0810 11:38:21.009118 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:21.009127 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:21.009290 (+ 163us) server_negotiation.cc:407] Connection header received
0810 11:38:21.009623 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:21.009633 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:21.009751 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:21.009886 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:21.011568 (+ 1682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.013107 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.016049 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.016953 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.017117 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.025404 (+ 8287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:39549: BlockingRecv error: recv got EOF from 127.3.240.193:39549 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 11:38:21.026342 17057 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 11:38:21.076121 17255 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33761
I20260810 11:38:21.076161 17461 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33761 every 8 connection(s)
I20260810 11:38:21.077593 17255 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:21.084952 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 17255
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:21.114974 17466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:21.086276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41326 (local address 127.3.240.254:39665)
0810 11:38:21.087014 (+ 738us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:21.087024 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:21.091599 (+ 4575us) server_negotiation.cc:407] Connection header received
0810 11:38:21.091799 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:21.091803 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:21.091900 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:21.092028 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:21.094973 (+ 2945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.095833 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.098523 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.099212 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.099350 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.103379 (+ 4029us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:21.103396 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:21.103428 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:21.103498 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:21.109932 (+ 6434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:21.111023 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:21.111038 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:21.111053 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:21.111367 (+ 314us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:21.111872 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:21.111880 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:21.111886 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:21.112813 (+ 927us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:21.113267 (+ 454us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:21.113652 (+ 385us) server_negotiation.cc:299] Negotiation successful
0810 11:38:21.114233 (+ 581us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":330,"threads_started":1}
W20260810 11:38:21.362527 17253 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.175s user 0.001s sys 0.013s
W20260810 11:38:21.364395 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:20.160187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:36203 (local address 127.3.240.254:39665)
0810 11:38:20.175527 (+ 15340us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:20.175538 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:20.177431 (+ 1893us) server_negotiation.cc:407] Connection header received
0810 11:38:20.181581 (+ 4150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:20.181585 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:20.181655 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:20.181805 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:20.183548 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.184324 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.185884 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.186294 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.186427 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.364109 (+1177682us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:36203: BlockingRecv error: recv got EOF from 127.3.240.194:36203 (error 108)
Metrics: {"server-negotiator.queue_time_us":15103,"thread_start_us":15344,"threads_started":1}
I20260810 11:38:21.365924 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:20.159794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:36203)
0810 11:38:20.176604 (+ 16810us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:20.176687 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:20.181070 (+ 4383us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:20.182100 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:20.182109 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:20.182675 (+ 566us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:20.183352 (+ 677us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:20.183370 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.184500 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.184505 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:20.185733 (+ 1228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:20.185745 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:20.186554 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:20.186562 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:20.186789 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:20.187570 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:20.187599 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:21.363951 (+1176352us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":16274,"thread_start_us":15994,"threads_started":1}
W20260810 11:38:21.367538 17251 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:39665 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:39665: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:21.382746 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:21.368678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:33659)
0810 11:38:21.369007 (+ 329us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:21.369039 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:21.369228 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:21.369726 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:21.369734 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:21.370538 (+ 804us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:21.370900 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.370918 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.372089 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.372099 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.374382 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.374406 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.375457 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.375467 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.375742 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.376746 (+ 1004us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:21.376783 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:21.382245 (+ 5462us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 11:38:21.382764 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:21.368873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:33659 (local address 127.3.240.254:39665)
0810 11:38:21.369059 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:21.369063 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:21.369149 (+ 86us) server_negotiation.cc:407] Connection header received
0810 11:38:21.369444 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:21.369449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:21.369537 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:21.369644 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:21.371111 (+ 1467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.371878 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.374591 (+ 2713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.375223 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.375331 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.382511 (+ 7180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:33659: BlockingRecv error: recv got EOF from 127.3.240.194:33659 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:21 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:21.395807 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:21.383721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:57019)
0810 11:38:21.383894 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:21.383913 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:21.384049 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:21.384765 (+ 716us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:21.384769 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:21.385073 (+ 304us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:21.385247 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.385255 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.387907 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.387913 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.389121 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.389136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.390727 (+ 1591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.390733 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.390880 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.391558 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:21.391577 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:21.395585 (+ 4008us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 11:38:21.396184 17251 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:38:21.396970 17254 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:21.384101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:57019 (local address 127.3.240.254:39665)
0810 11:38:21.384341 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:21.384351 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:21.384376 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:38:21.384450 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:21.384458 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:21.384563 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:21.384710 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:21.385412 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.387509 (+ 2097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.389314 (+ 1805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.390452 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.390665 (+ 213us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.396549 (+ 5884us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:57019: BlockingRecv error: recv got EOF from 127.3.240.194:57019 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:22.039594 17474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:22.027370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:58071)
0810 11:38:22.028137 (+ 767us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:22.028152 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:22.028316 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:22.028695 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:22.028700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:22.029149 (+ 449us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:22.029317 (+ 168us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.029326 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.030357 (+ 1031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.030364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.031379 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.031391 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.032279 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.032286 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.032564 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.033472 (+ 908us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:22.033491 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:22.039234 (+ 5743us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":657,"thread_start_us":515,"threads_started":1}
W20260810 11:38:22.039933 17473 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:22.027404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:58071 (local address 127.3.240.254:39665)
0810 11:38:22.027991 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:22.027998 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:22.028246 (+ 248us) server_negotiation.cc:407] Connection header received
0810 11:38:22.028453 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:22.028459 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:22.028544 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:22.028658 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:22.029467 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.030206 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.031520 (+ 1314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.032120 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.032266 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.039448 (+ 7182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:58071: BlockingRecv error: recv got EOF from 127.3.240.193:58071 (error 108)
Metrics: {"server-negotiator.queue_time_us":427,"thread_start_us":284,"threads_started":1}
W20260810 11:38:22.247972 17465 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.151s user 0.003s sys 0.013s
W20260810 11:38:22.249415 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:21.081121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44205 (local address 127.3.240.254:39665)
0810 11:38:21.081393 (+ 272us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:21.081399 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:21.083636 (+ 2237us) server_negotiation.cc:407] Connection header received
0810 11:38:21.088624 (+ 4988us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:21.088634 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:21.088748 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:21.088908 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:21.091896 (+ 2988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.092755 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.094328 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.094760 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.094902 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.249101 (+1154199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:44205: BlockingRecv error: recv got EOF from 127.3.240.195:44205 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:38:22.249745 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:21.081602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:44205)
0810 11:38:21.083396 (+ 1794us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:21.083468 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:21.088184 (+ 4716us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:21.089299 (+ 1115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:21.089317 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:21.090415 (+ 1098us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:21.091637 (+ 1222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.091675 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.092928 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.092935 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.094176 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:21.094191 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:21.096038 (+ 1847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:21.096043 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:21.096242 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:21.096966 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:21.096989 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:22.248766 (+1151777us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1286,"thread_start_us":770,"threads_started":1}
W20260810 11:38:22.250818 17463 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:39665 (0 consecutive failures): Not authorized: Failed to ping master at 127.3.240.254:39665: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:22.262022 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:22.251586 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:46401)
0810 11:38:22.251802 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:22.251820 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:22.251917 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:22.252361 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:22.252364 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:22.252862 (+ 498us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:22.253043 (+ 181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.253053 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.254209 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.254216 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.255238 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.255250 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.256037 (+ 787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.256043 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.256208 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.256868 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:22.256886 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:22.261704 (+ 4818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 11:38:22.262209 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:22.251784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46401 (local address 127.3.240.254:39665)
0810 11:38:22.252034 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:22.252039 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:22.252057 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:38:22.252109 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:22.252113 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:22.252187 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:22.252285 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:22.253219 (+ 934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.254056 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.255385 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.255837 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.255982 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.261927 (+ 5945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:46401: BlockingRecv error: recv got EOF from 127.3.240.195:46401 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:22.272022 17465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:22.262851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:47219)
0810 11:38:22.263122 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:22.263144 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:22.263271 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:22.263602 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:22.263606 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:22.263987 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:22.264142 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.264150 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.265173 (+ 1023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.265178 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.266136 (+ 958us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.266147 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.266869 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.266875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.267090 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.267694 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:22.267711 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:22.271643 (+ 3932us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":134}
W20260810 11:38:22.272253 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:22.262925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47219 (local address 127.3.240.254:39665)
0810 11:38:22.263127 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:22.263134 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:22.263213 (+ 79us) server_negotiation.cc:407] Connection header received
0810 11:38:22.263388 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:22.263393 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:22.263465 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:22.263566 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:22.264283 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.265033 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.266275 (+ 1242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.266713 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.266843 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.271971 (+ 5128us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:47219: BlockingRecv error: recv got EOF from 127.3.240.195:47219 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260810 11:38:22.272501 17463 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:22 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:22.409577 17476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:22.396840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:34787)
0810 11:38:22.397864 (+ 1024us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:22.397881 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:22.398006 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:22.398473 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:22.398478 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:22.399137 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:22.399350 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.399361 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.400695 (+ 1334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.400701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.401775 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:22.401787 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.402859 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.402874 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:22.403162 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.403985 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:22.404014 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:22.409235 (+ 5221us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":892,"thread_start_us":618,"threads_started":1}
W20260810 11:38:22.410530 17059 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:22.397050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:34787 (local address 127.3.240.254:39665)
0810 11:38:22.397247 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:22.397252 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:22.398006 (+ 754us) server_negotiation.cc:407] Connection header received
0810 11:38:22.398121 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:22.398125 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:22.398208 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:22.398294 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:22.399629 (+ 1335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.400519 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.401950 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:22.402562 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:22.402773 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:22.410215 (+ 7442us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:34787: BlockingRecv error: recv got EOF from 127.3.240.194:34787 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:23.051254 17477 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:23.040585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:57157)
0810 11:38:23.041139 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:23.041156 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:23.041277 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:23.041639 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:23.041643 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:23.042041 (+ 398us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:23.042214 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.042222 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.043510 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.043514 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.044660 (+ 1146us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.044674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.045424 (+ 750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.045429 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.045567 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.046252 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:23.046271 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:23.050753 (+ 4482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":303,"threads_started":1}
W20260810 11:38:23.051343 17478 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:23.040731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57157 (local address 127.3.240.254:39665)
0810 11:38:23.041270 (+ 539us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:23.041274 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:23.041291 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:38:23.041381 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:23.041385 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:23.041454 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:23.041549 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:23.042388 (+ 839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.043333 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.044829 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.045256 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.045362 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.051024 (+ 5662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:57157: BlockingRecv error: recv got EOF from 127.3.240.193:57157 (error 108)
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":319,"threads_started":1}
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:23.286221 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:23.273483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:57957)
0810 11:38:23.274369 (+ 886us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:23.274389 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:23.274613 (+ 224us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:23.275101 (+ 488us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:23.275108 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:23.275751 (+ 643us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:23.275943 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.275954 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.277238 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.277251 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.278660 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.278678 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.279746 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.279752 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.279899 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.280774 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:23.280796 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:23.285800 (+ 5004us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":523,"threads_started":1}
W20260810 11:38:23.286507 17478 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:23.273717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57957 (local address 127.3.240.254:39665)
0810 11:38:23.274064 (+ 347us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:23.274073 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:23.274514 (+ 441us) server_negotiation.cc:407] Connection header received
0810 11:38:23.274784 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:23.274789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:23.274893 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:23.275048 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:23.276115 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.277080 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.278894 (+ 1814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.279513 (+ 619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.279638 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.286225 (+ 6587us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:57957: BlockingRecv error: recv got EOF from 127.3.240.195:57957 (error 108)
Metrics: {"server-negotiator.queue_time_us":116}
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:23 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:23.425669 17482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:23.414843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:52917)
0810 11:38:23.415556 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:23.415578 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:23.415738 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:23.416204 (+ 466us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:23.416209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:23.416796 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:23.417047 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.417057 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.418307 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.418312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.419361 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:23.419374 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.420484 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.420491 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:23.420679 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.421453 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:23.421479 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:23.425406 (+ 3927us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":424,"threads_started":1}
W20260810 11:38:23.425982 17478 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:23.415051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52917 (local address 127.3.240.254:39665)
0810 11:38:23.415365 (+ 314us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:23.415372 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:23.415715 (+ 343us) server_negotiation.cc:407] Connection header received
0810 11:38:23.415872 (+ 157us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:23.415876 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:23.415963 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:23.416101 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:23.417261 (+ 1160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.418100 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.419545 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:23.420292 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:23.420462 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:23.425661 (+ 5199us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52917: BlockingRecv error: recv got EOF from 127.3.240.194:52917 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:24.064708 17483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:24.052598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:51309)
0810 11:38:24.053572 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:24.053608 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:24.053835 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:24.054333 (+ 498us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:24.054339 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:24.055212 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:24.055434 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.055445 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.056764 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.056769 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.057778 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.057791 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.058738 (+ 947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.058747 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.059097 (+ 350us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.060009 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:24.060059 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:24.064447 (+ 4388us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":553,"threads_started":1}
W20260810 11:38:24.065143 17484 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:24.052992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51309 (local address 127.3.240.254:39665)
0810 11:38:24.053964 (+ 972us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:24.053971 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:24.054003 (+ 32us) server_negotiation.cc:407] Connection header received
0810 11:38:24.054063 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:24.054067 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:24.054145 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:24.054291 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:24.055618 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.056575 (+ 957us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.057929 (+ 1354us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.058456 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.058595 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.064719 (+ 6124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:51309: BlockingRecv error: recv got EOF from 127.3.240.193:51309 (error 108)
Metrics: {"server-negotiator.queue_time_us":819,"thread_start_us":626,"threads_started":1}
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:24.299304 17485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:24.287392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:38747)
0810 11:38:24.287946 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:24.287964 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:24.288103 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:24.288484 (+ 381us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:24.288488 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:24.289036 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:24.289260 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.289271 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.290470 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.290475 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.291651 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.291667 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.292649 (+ 982us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.292658 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.292841 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.293637 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:24.293669 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:24.299039 (+ 5370us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":442,"thread_start_us":340,"threads_started":1}
W20260810 11:38:24.299805 17484 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:24.287522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38747 (local address 127.3.240.254:39665)
0810 11:38:24.287833 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:24.287839 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:24.288093 (+ 254us) server_negotiation.cc:407] Connection header received
0810 11:38:24.288198 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:24.288212 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:24.288297 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:24.288384 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:24.289418 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.290282 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.291903 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.292441 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.292550 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.299501 (+ 6951us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:38747: BlockingRecv error: recv got EOF from 127.3.240.195:38747 (error 108)
Metrics: {"server-negotiator.queue_time_us":119}
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:24 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:24.439195 17486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:24.427104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:52351)
0810 11:38:24.428086 (+ 982us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:24.428106 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:24.428249 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:24.428667 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:24.428671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:24.429098 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:24.429286 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.429296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.430493 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.430497 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.431631 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:24.431645 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.432806 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.432811 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:24.432958 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.433729 (+ 771us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:24.433756 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:24.438727 (+ 4971us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":828,"thread_start_us":616,"threads_started":1}
W20260810 11:38:24.439653 17484 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:24.427249 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52351 (local address 127.3.240.254:39665)
0810 11:38:24.427519 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:24.427539 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:24.428202 (+ 663us) server_negotiation.cc:407] Connection header received
0810 11:38:24.428423 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:24.428427 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:24.428513 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:24.428603 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:24.429464 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.430322 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.431876 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:24.432627 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:24.432760 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:24.439359 (+ 6599us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52351: BlockingRecv error: recv got EOF from 127.3.240.194:52351 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:25.076429 17489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:25.065911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:46635)
0810 11:38:25.066520 (+ 609us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:25.066537 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:25.066642 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:25.067084 (+ 442us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:25.067087 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:25.067472 (+ 385us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:25.067641 (+ 169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.067649 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.068865 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.068869 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.070104 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.070117 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.070901 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.070906 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.071145 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.071832 (+ 687us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:25.071851 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:25.076057 (+ 4206us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":494,"thread_start_us":343,"threads_started":1}
W20260810 11:38:25.076581 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:25.066005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46635 (local address 127.3.240.254:39665)
0810 11:38:25.066562 (+ 557us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:25.066567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:25.066585 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:38:25.066813 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:25.066818 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:25.066893 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:25.067024 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:25.067820 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.068577 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.070301 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.070739 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.070890 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.076291 (+ 5401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:46635: BlockingRecv error: recv got EOF from 127.3.240.193:46635 (error 108)
Metrics: {"server-negotiator.queue_time_us":458,"thread_start_us":339,"threads_started":1}
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:25.311947 17491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:25.300641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:38389)
0810 11:38:25.301330 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:25.301349 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:25.301479 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:25.301910 (+ 431us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:25.301914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:25.302346 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:25.302574 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.302585 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.303843 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.303847 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.305036 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.305060 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.305981 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.305986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.306150 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.306817 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:25.306837 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:25.311564 (+ 4727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":358,"threads_started":1}
W20260810 11:38:25.312521 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:25.300731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38389 (local address 127.3.240.254:39665)
0810 11:38:25.300999 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:25.301003 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:25.301496 (+ 493us) server_negotiation.cc:407] Connection header received
0810 11:38:25.301586 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:25.301589 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:25.301691 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:25.301819 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:25.302764 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.303693 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.305262 (+ 1569us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.305796 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.305916 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.311983 (+ 6067us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:38389: BlockingRecv error: recv got EOF from 127.3.240.195:38389 (error 108)
Metrics: {"server-negotiator.queue_time_us":137}
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:25 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:25.452788 17494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:25.440766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:55719)
0810 11:38:25.441492 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:25.441511 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:25.441651 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:25.442136 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:25.442142 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:25.442996 (+ 854us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:25.443214 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.443224 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.444640 (+ 1416us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.444645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.445689 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:25.445702 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.446531 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.446536 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:25.446672 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.447416 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:25.447436 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:25.452389 (+ 4953us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":421,"threads_started":1}
W20260810 11:38:25.453035 17490 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:25.440824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:55719 (local address 127.3.240.254:39665)
0810 11:38:25.441023 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:25.441028 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:25.441676 (+ 648us) server_negotiation.cc:407] Connection header received
0810 11:38:25.441788 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:25.441792 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:25.441899 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:25.442002 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:25.443412 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.444462 (+ 1050us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.445882 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:25.446339 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:25.446446 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:25.452720 (+ 6274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:55719: BlockingRecv error: recv got EOF from 127.3.240.194:55719 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:26.093605 17495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:26.077852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:33023)
0810 11:38:26.079213 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:26.079235 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:26.079404 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:26.079940 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:26.079945 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:26.080470 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:26.080782 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.080793 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.082037 (+ 1244us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.082041 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.083231 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.083246 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.084380 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.084391 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.084616 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.089423 (+ 4807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:26.089445 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:26.093138 (+ 3693us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":1202,"thread_start_us":1072,"threads_started":1}
W20260810 11:38:26.093854 17497 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:26.078127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:33023 (local address 127.3.240.254:39665)
0810 11:38:26.078830 (+ 703us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:26.078839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:26.079362 (+ 523us) server_negotiation.cc:407] Connection header received
0810 11:38:26.079549 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:26.079557 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:26.079690 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:26.079812 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:26.081031 (+ 1219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.081865 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.083476 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.084192 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.084298 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.093480 (+ 9182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:33023: BlockingRecv error: recv got EOF from 127.3.240.193:33023 (error 108)
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":336,"threads_started":1}
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:26.326581 17498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:26.314400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:37379)
0810 11:38:26.315463 (+ 1063us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:26.315482 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:26.315606 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:26.315972 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:26.315975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:26.316409 (+ 434us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:26.316591 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.316600 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.318010 (+ 1410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.318014 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.319290 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.319304 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.320208 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.320214 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.320356 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.321190 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:26.321225 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:26.326137 (+ 4912us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":941,"thread_start_us":639,"threads_started":1}
W20260810 11:38:26.326797 17497 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:26.314497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37379 (local address 127.3.240.254:39665)
0810 11:38:26.314806 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:26.314811 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:26.315597 (+ 786us) server_negotiation.cc:407] Connection header received
0810 11:38:26.315715 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:26.315723 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:26.315791 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:26.315885 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:26.316805 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.317819 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.319522 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.320005 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.320176 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.326458 (+ 6282us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:37379: BlockingRecv error: recv got EOF from 127.3.240.195:37379 (error 108)
Metrics: {"server-negotiator.queue_time_us":142}
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:26 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:26.468865 17502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:26.454350 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:49195)
0810 11:38:26.455137 (+ 787us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:26.455156 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:26.455374 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:26.456723 (+ 1349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:26.456730 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:26.457344 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:26.457604 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.457619 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.459542 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.459557 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.461426 (+ 1869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:26.461464 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.462618 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.462624 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:26.462829 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.463538 (+ 709us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:26.463565 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:26.468476 (+ 4911us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":650,"thread_start_us":436,"threads_started":1}
W20260810 11:38:26.468940 17497 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:26.454538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49195 (local address 127.3.240.254:39665)
0810 11:38:26.454856 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:26.454864 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:26.455264 (+ 400us) server_negotiation.cc:407] Connection header received
0810 11:38:26.455651 (+ 387us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:26.455657 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:26.456467 (+ 810us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:26.456622 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:26.457862 (+ 1240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.459246 (+ 1384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.461608 (+ 2362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:26.462426 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:26.462576 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:26.468673 (+ 6097us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:49195: BlockingRecv error: recv got EOF from 127.3.240.194:49195 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:27.104724 17504 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:27.094894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:47473)
0810 11:38:27.095474 (+ 580us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:27.095497 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:27.095607 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:27.095952 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:27.095956 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:27.096407 (+ 451us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:27.096583 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.096591 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.097914 (+ 1323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.097919 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.098855 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.098867 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.099684 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.099689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.099822 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.100481 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:27.100501 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:27.104378 (+ 3877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":305,"threads_started":1}
W20260810 11:38:27.105439 17505 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:27.095027 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47473 (local address 127.3.240.254:39665)
0810 11:38:27.095565 (+ 538us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:27.095569 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:27.095585 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:38:27.095715 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:27.095719 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:27.095788 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:27.095876 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:27.096758 (+ 882us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.097618 (+ 860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.099090 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.099496 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.099604 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.105013 (+ 5409us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:47473: BlockingRecv error: recv got EOF from 127.3.240.193:47473 (error 108)
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":318,"threads_started":1}
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:27.340149 17506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:27.327894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:45875)
0810 11:38:27.328618 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:27.328633 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:27.328805 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:27.329150 (+ 345us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:27.329154 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:27.329645 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:27.329795 (+ 150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.329803 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.331102 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.331107 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.332256 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.332273 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.333557 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.333565 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.333797 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.334974 (+ 1177us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:27.334997 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:27.339902 (+ 4905us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":626,"thread_start_us":485,"threads_started":1}
W20260810 11:38:27.340374 17505 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:27.328125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:45875 (local address 127.3.240.254:39665)
0810 11:38:27.328471 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:27.328478 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:27.328719 (+ 241us) server_negotiation.cc:407] Connection header received
0810 11:38:27.328939 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:27.328948 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:27.329010 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:27.329118 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:27.329941 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.330865 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.332576 (+ 1711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.333294 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.333405 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.340132 (+ 6727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:45875: BlockingRecv error: recv got EOF from 127.3.240.195:45875 (error 108)
Metrics: {"server-negotiator.queue_time_us":175}
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:27 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:27.481251 17507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:27.470086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:52701)
0810 11:38:27.470638 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:27.470655 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:27.470782 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:27.471344 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:27.471349 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:27.471781 (+ 432us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:27.471980 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.471989 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.473158 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.473163 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.474597 (+ 1434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:27.474630 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.475850 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.475855 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:27.476007 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.476807 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:27.476825 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:27.480993 (+ 4168us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":285,"threads_started":1}
W20260810 11:38:27.481498 17505 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:27.470172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52701 (local address 127.3.240.254:39665)
0810 11:38:27.470468 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:27.470477 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:27.470814 (+ 337us) server_negotiation.cc:407] Connection header received
0810 11:38:27.470919 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:27.470989 (+ 70us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:27.471142 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:27.471270 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:27.472159 (+ 889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.472978 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.474844 (+ 1866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:27.475647 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:27.475768 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:27.481212 (+ 5444us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:52701: BlockingRecv error: recv got EOF from 127.3.240.194:52701 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:28.118124 17509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:28.106128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:49827)
0810 11:38:28.107188 (+ 1060us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:28.107205 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:28.107387 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:28.107767 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:28.107772 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:28.108255 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:28.108459 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.108467 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.109522 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.109527 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.110493 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.110504 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.111353 (+ 849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.111360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.111524 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.112805 (+ 1281us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:28.112844 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:28.117759 (+ 4915us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":955,"thread_start_us":742,"threads_started":1}
W20260810 11:38:28.118342 17508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:28.106222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49827 (local address 127.3.240.254:39665)
0810 11:38:28.106801 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:28.106806 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:28.107302 (+ 496us) server_negotiation.cc:407] Connection header received
0810 11:38:28.107522 (+ 220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:28.107531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:28.107612 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:28.107734 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:28.108610 (+ 876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.109381 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.110651 (+ 1270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.111161 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.111301 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.117987 (+ 6686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:49827: BlockingRecv error: recv got EOF from 127.3.240.193:49827 (error 108)
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":331,"threads_started":1}
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:28.352080 17510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:28.341283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:53789)
0810 11:38:28.341839 (+ 556us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:28.341855 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:28.342020 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:28.342359 (+ 339us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:28.342363 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:28.342809 (+ 446us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:28.343082 (+ 273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.343093 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.344515 (+ 1422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.344522 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.345712 (+ 1190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.345726 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.346488 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.346493 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.346626 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.347406 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:28.347425 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:28.351529 (+ 4104us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":331,"threads_started":1}
W20260810 11:38:28.352171 17508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:28.341370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:53789 (local address 127.3.240.254:39665)
0810 11:38:28.341662 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:28.341670 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:28.341950 (+ 280us) server_negotiation.cc:407] Connection header received
0810 11:38:28.342147 (+ 197us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:28.342152 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:28.342230 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:28.342331 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:28.343246 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.344112 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.345888 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.346288 (+ 400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.346394 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.351901 (+ 5507us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:53789: BlockingRecv error: recv got EOF from 127.3.240.195:53789 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:28 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:28.492628 17512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:28.482402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:41527)
0810 11:38:28.483089 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:28.483106 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:28.483225 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:28.483597 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:28.483601 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:28.484017 (+ 416us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:28.484199 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.484208 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.485289 (+ 1081us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.485293 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.486301 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:28.486314 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.487140 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.487154 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:28.487335 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.488069 (+ 734us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:28.488088 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:28.492189 (+ 4101us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":340,"threads_started":1}
W20260810 11:38:28.493013 17508 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:28.482494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:41527 (local address 127.3.240.254:39665)
0810 11:38:28.482760 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:28.482765 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:28.483233 (+ 468us) server_negotiation.cc:407] Connection header received
0810 11:38:28.483352 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:28.483355 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:28.483421 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:28.483500 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:28.484363 (+ 863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.485127 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.486470 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:28.486923 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:28.487095 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:28.492619 (+ 5524us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:41527: BlockingRecv error: recv got EOF from 127.3.240.194:41527 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:29.130179 17513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:29.119323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:47575)
0810 11:38:29.119963 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:29.119980 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:29.120176 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:29.120708 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:29.120711 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:29.121188 (+ 477us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:29.121484 (+ 296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.121495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.122991 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.122997 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.123962 (+ 965us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.123974 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.124978 (+ 1004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.124983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.125104 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.125761 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:29.125781 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:29.129795 (+ 4014us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":349,"threads_started":1}
W20260810 11:38:29.130452 17514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:29.119585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47575 (local address 127.3.240.254:39665)
0810 11:38:29.120275 (+ 690us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:29.120281 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:29.120302 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:38:29.120377 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:29.120381 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:29.120473 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:29.120627 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:29.121676 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.122786 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.124172 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.124740 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.124888 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.130164 (+ 5276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:47575: BlockingRecv error: recv got EOF from 127.3.240.193:47575 (error 108)
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":429,"threads_started":1}
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:29.365053 17515 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:29.353507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:54865)
0810 11:38:29.354292 (+ 785us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:29.354311 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:29.354444 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:29.354966 (+ 522us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:29.354971 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:29.355503 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:29.355755 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.355766 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.357207 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.357216 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.358762 (+ 1546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.358777 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.359638 (+ 861us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.359647 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.359794 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.360531 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:29.360550 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:29.364785 (+ 4235us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":387,"threads_started":1}
W20260810 11:38:29.365252 17514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:29.353800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:54865 (local address 127.3.240.254:39665)
0810 11:38:29.354158 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:29.354179 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:29.354389 (+ 210us) server_negotiation.cc:407] Connection header received
0810 11:38:29.354595 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:29.354599 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:29.354672 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:29.354769 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:29.355958 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.356943 (+ 985us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.358976 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.359431 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.359540 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.364996 (+ 5456us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:54865: BlockingRecv error: recv got EOF from 127.3.240.195:54865 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:29 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:29.506130 17517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:29.494089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:60723)
0810 11:38:29.494965 (+ 876us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:29.494984 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:29.495120 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:29.495647 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:29.495653 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:29.496384 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:29.496582 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.496591 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.498360 (+ 1769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.498365 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.499532 (+ 1167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:29.499548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.500527 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.500532 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:29.500855 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.501657 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:29.501681 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:29.505839 (+ 4158us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":686,"thread_start_us":522,"threads_started":1}
W20260810 11:38:29.506362 17514 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:29.494245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60723 (local address 127.3.240.254:39665)
0810 11:38:29.494471 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:29.494476 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:29.495145 (+ 669us) server_negotiation.cc:407] Connection header received
0810 11:38:29.495281 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:29.495287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:29.495406 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:29.495553 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:29.496847 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.498159 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.499763 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:29.500278 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:29.500401 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:29.506066 (+ 5665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:60723: BlockingRecv error: recv got EOF from 127.3.240.194:60723 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:30.143934 17518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:30.131391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:60887)
0810 11:38:30.132132 (+ 741us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:30.132160 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:30.132271 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:30.132675 (+ 404us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:30.132680 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:30.133183 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:30.133347 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.133356 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.134841 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.134847 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.135875 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.135887 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.136792 (+ 905us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.136798 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.137041 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.137672 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:30.137690 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:30.143484 (+ 5794us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":300,"threads_started":1}
W20260810 11:38:30.144241 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:30.131469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:60887 (local address 127.3.240.254:39665)
0810 11:38:30.132179 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:30.132185 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:30.132213 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:38:30.132416 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:30.132421 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:30.132510 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:30.132646 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:30.133494 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.134676 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.136036 (+ 1360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.136624 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.136794 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.143857 (+ 7063us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:60887: BlockingRecv error: recv got EOF from 127.3.240.193:60887 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":421,"threads_started":1}
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:30.377832 17521 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:30.366126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:55965)
0810 11:38:30.366757 (+ 631us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:30.366773 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:30.366972 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:30.367313 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:30.367318 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:30.367777 (+ 459us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:30.367957 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.367966 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.369148 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.369153 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.370216 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.370228 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.371264 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.371271 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.371434 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.372579 (+ 1145us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:30.372606 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:30.377388 (+ 4782us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":360,"threads_started":1}
W20260810 11:38:30.377951 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:30.366285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55965 (local address 127.3.240.254:39665)
0810 11:38:30.366559 (+ 274us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:30.366567 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:30.366867 (+ 300us) server_negotiation.cc:407] Connection header received
0810 11:38:30.367111 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:30.367116 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:30.367178 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:30.367283 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:30.368102 (+ 819us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.368991 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.370365 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.371053 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.371195 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.377680 (+ 6485us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:55965: BlockingRecv error: recv got EOF from 127.3.240.195:55965 (error 108)
Metrics: {"server-negotiator.queue_time_us":100}
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:30 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:30.520712 17522 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:30.507837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:39869)
0810 11:38:30.508545 (+ 708us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:30.508564 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:30.508707 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:30.509193 (+ 486us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:30.509198 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:30.509695 (+ 497us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:30.509871 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.509880 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.511287 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.511293 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.512644 (+ 1351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:30.512660 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.514004 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.514011 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:30.514192 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.515162 (+ 970us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:30.515195 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:30.520406 (+ 5211us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":412,"threads_started":1}
W20260810 11:38:30.520991 17519 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:30.507896 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:39869 (local address 127.3.240.254:39665)
0810 11:38:30.508147 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:30.508152 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:30.508700 (+ 548us) server_negotiation.cc:407] Connection header received
0810 11:38:30.508834 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:30.508837 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:30.508926 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:30.509032 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:30.510041 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.511023 (+ 982us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.512809 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:30.513735 (+ 926us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:30.513901 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:30.520626 (+ 6725us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:39869: BlockingRecv error: recv got EOF from 127.3.240.194:39869 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:31.159453 17523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:31.145484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:58521)
0810 11:38:31.146036 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:31.146062 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:31.146205 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:31.146758 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:31.146763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:31.147207 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:31.147372 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.147380 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.148455 (+ 1075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.148460 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.149429 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.149440 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.150169 (+ 729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.150175 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.150357 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.153987 (+ 3630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:31.154008 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:31.158827 (+ 4819us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":400,"thread_start_us":288,"threads_started":1}
W20260810 11:38:31.159715 17524 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:31.145583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:58521 (local address 127.3.240.254:39665)
0810 11:38:31.146324 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:31.146330 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:31.146346 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:38:31.146404 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:31.146408 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:31.146576 (+ 168us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:31.146722 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:31.147535 (+ 813us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.148297 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.149577 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.150012 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.150147 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.159403 (+ 9256us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:58521: BlockingRecv error: recv got EOF from 127.3.240.193:58521 (error 108)
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":473,"threads_started":1}
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:31.389348 17526 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:31.379309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:49309)
0810 11:38:31.379871 (+ 562us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:31.379896 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:31.380020 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:31.380394 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:31.380397 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:31.380822 (+ 425us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:31.381009 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.381018 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.382150 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.382154 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.383138 (+ 984us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.383150 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.383895 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.383900 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.384048 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.384759 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:31.384792 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:31.389080 (+ 4288us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":305,"threads_started":1}
W20260810 11:38:31.389539 17524 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:31.379458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:49309 (local address 127.3.240.254:39665)
0810 11:38:31.379729 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:31.379735 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:31.380034 (+ 299us) server_negotiation.cc:407] Connection header received
0810 11:38:31.380123 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:31.380127 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:31.380208 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:31.380322 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:31.381195 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.381997 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.383318 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.383727 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.383828 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.389287 (+ 5459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:49309: BlockingRecv error: recv got EOF from 127.3.240.195:49309 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:31 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:31.533687 17527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:31.521971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:49009)
0810 11:38:31.522965 (+ 994us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:31.522983 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:31.523154 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:31.523538 (+ 384us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:31.523543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:31.524082 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:31.524268 (+ 186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.524277 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.525425 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.525431 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.526431 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:31.526442 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.527231 (+ 789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.527237 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:31.527412 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.528078 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:31.528097 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:31.533261 (+ 5164us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":845,"thread_start_us":496,"threads_started":1}
W20260810 11:38:31.533756 17524 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:31.522076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49009 (local address 127.3.240.254:39665)
0810 11:38:31.522546 (+ 470us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:31.522555 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:31.523084 (+ 529us) server_negotiation.cc:407] Connection header received
0810 11:38:31.523296 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:31.523307 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:31.523392 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:31.523508 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:31.524422 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.525277 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.526580 (+ 1303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:31.527066 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:31.527196 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:31.533471 (+ 6275us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:49009: BlockingRecv error: recv got EOF from 127.3.240.194:49009 (error 108)
Metrics: {"server-negotiator.queue_time_us":171}
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:32.171131 17528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:32.160745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:57633)
0810 11:38:32.161366 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:32.161392 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:32.161546 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:32.162102 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:32.162106 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:32.162561 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:32.162754 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.162763 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.163950 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.163954 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.164921 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.164933 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.165846 (+ 913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.165851 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.165974 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.166604 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:32.166623 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:32.170741 (+ 4118us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":480,"thread_start_us":363,"threads_started":1}
W20260810 11:38:32.171329 17530 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:32.160999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57633 (local address 127.3.240.254:39665)
0810 11:38:32.161725 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:32.161731 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:32.161750 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:38:32.161813 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:32.161817 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:32.161902 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:32.162043 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:32.162994 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.163787 (+ 793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.165095 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.165576 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.165688 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.171003 (+ 5315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:57633: BlockingRecv error: recv got EOF from 127.3.240.193:57633 (error 108)
Metrics: {"server-negotiator.queue_time_us":595,"thread_start_us":387,"threads_started":1}
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:32.400851 17531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:32.390520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:52193)
0810 11:38:32.391099 (+ 579us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:32.391116 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:32.391277 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:32.391624 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:32.391628 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:32.392056 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:32.392220 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.392229 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.393300 (+ 1071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.393306 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.394280 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.394292 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.395034 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.395039 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.395192 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.395802 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:32.395819 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:32.400420 (+ 4601us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":299,"threads_started":1}
W20260810 11:38:32.401051 17530 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:32.390588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:52193 (local address 127.3.240.254:39665)
0810 11:38:32.390811 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:32.390819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:32.391206 (+ 387us) server_negotiation.cc:407] Connection header received
0810 11:38:32.391407 (+ 201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:32.391414 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:32.391486 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:32.391595 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:32.392381 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.393151 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.394424 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.394833 (+ 409us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.394995 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.400722 (+ 5727us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:52193: BlockingRecv error: recv got EOF from 127.3.240.195:52193 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:32 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:32.545346 17532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:32.534996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:60383)
0810 11:38:32.535947 (+ 951us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:32.535975 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:32.536170 (+ 195us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:32.536674 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:32.536679 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:32.537162 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:32.537339 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.537348 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.538433 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.538441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.539593 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:32.539606 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.540488 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.540498 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:32.540696 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.541397 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:32.541416 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:32.545108 (+ 3692us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":760,"thread_start_us":533,"threads_started":1}
W20260810 11:38:32.545800 17530 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:32.535186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60383 (local address 127.3.240.254:39665)
0810 11:38:32.535560 (+ 374us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:32.535571 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:32.536133 (+ 562us) server_negotiation.cc:407] Connection header received
0810 11:38:32.536396 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:32.536401 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:32.536509 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:32.536639 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:32.537527 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.538289 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.539737 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:32.540199 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:32.540358 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:32.545357 (+ 4999us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:60383: BlockingRecv error: recv got EOF from 127.3.240.194:60383 (error 108)
Metrics: {"server-negotiator.queue_time_us":107}
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:33.184264 17535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:33.172366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:59375)
0810 11:38:33.173062 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:33.173082 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:33.173242 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:33.173757 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:33.173765 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:33.174471 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:33.174786 (+ 315us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.174802 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.176178 (+ 1376us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.176183 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.177297 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.177311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.178200 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.178206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.178372 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.179146 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:33.179166 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:33.183979 (+ 4813us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":405,"threads_started":1}
W20260810 11:38:33.184955 17536 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:33.172664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:59375 (local address 127.3.240.254:39665)
0810 11:38:33.173294 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:33.173299 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:33.173318 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:38:33.173402 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:33.173406 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:33.173514 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:33.173649 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:33.175034 (+ 1385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.175888 (+ 854us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.177481 (+ 1593us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.177972 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.178102 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.184645 (+ 6543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:59375: BlockingRecv error: recv got EOF from 127.3.240.193:59375 (error 108)
Metrics: {"server-negotiator.queue_time_us":496,"thread_start_us":365,"threads_started":1}
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:33.413327 17539 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:33.402363 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:46947)
0810 11:38:33.403049 (+ 686us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:33.403068 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:33.403268 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:33.403758 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:33.403766 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:33.404466 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:33.404735 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.404751 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.406000 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.406006 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.407006 (+ 1000us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.407022 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.407739 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.407745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.407945 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.408576 (+ 631us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:33.408595 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:33.413085 (+ 4490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":372,"threads_started":1}
W20260810 11:38:33.413619 17536 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:33.402438 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:46947 (local address 127.3.240.254:39665)
0810 11:38:33.402683 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:33.402689 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:33.403182 (+ 493us) server_negotiation.cc:407] Connection header received
0810 11:38:33.403408 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:33.403414 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:33.403502 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:33.403676 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:33.404924 (+ 1248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.405855 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.407160 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.407573 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.407713 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.413334 (+ 5621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:46947: BlockingRecv error: recv got EOF from 127.3.240.195:46947 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:33 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:33.557734 17541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:33.546334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:41253)
0810 11:38:33.546904 (+ 570us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:33.546920 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:33.547072 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:33.547495 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:33.547498 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:33.547887 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:33.548057 (+ 170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.548065 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.549157 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.549162 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.550106 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:33.550117 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.551109 (+ 992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.551115 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:33.551656 (+ 541us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.552579 (+ 923us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:33.552604 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:33.557422 (+ 4818us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":325,"threads_started":1}
W20260810 11:38:33.558174 17536 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:33.546476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:41253 (local address 127.3.240.254:39665)
0810 11:38:33.546685 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:33.546690 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:33.547031 (+ 341us) server_negotiation.cc:407] Connection header received
0810 11:38:33.547247 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:33.547251 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:33.547322 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:33.547414 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:33.548212 (+ 798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.548991 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.550282 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:33.550713 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:33.551020 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:33.557842 (+ 6822us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:41253: BlockingRecv error: recv got EOF from 127.3.240.194:41253 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:34.196631 17544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:34.185416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:41009)
0810 11:38:34.186087 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:34.186104 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:34.186227 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:34.186656 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:34.186659 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:34.187103 (+ 444us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:34.187251 (+ 148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.187260 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.188857 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.188862 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.189831 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.189852 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.190719 (+ 867us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.190723 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.190867 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.191631 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:34.191650 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:34.196354 (+ 4704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":406,"threads_started":1}
W20260810 11:38:34.197108 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:34.185502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:41009 (local address 127.3.240.254:39665)
0810 11:38:34.186152 (+ 650us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:34.186157 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:34.186173 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:38:34.186372 (+ 199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:34.186376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:34.186473 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:34.186568 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:34.187461 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.188628 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.190030 (+ 1402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.190567 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.190684 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.196670 (+ 5986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:41009: BlockingRecv error: recv got EOF from 127.3.240.193:41009 (error 108)
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":305,"threads_started":1}
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:34.427291 17549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:34.414689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:37667)
0810 11:38:34.415458 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:34.415478 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:34.415675 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:34.416041 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:34.416046 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:34.416528 (+ 482us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:34.416857 (+ 329us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.416881 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.418158 (+ 1277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.418164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.419920 (+ 1756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.419938 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.421128 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.421135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.421421 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.422296 (+ 875us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:34.422318 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:34.426678 (+ 4360us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":434,"threads_started":1}
W20260810 11:38:34.427425 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:34.414799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:37667 (local address 127.3.240.254:39665)
0810 11:38:34.415161 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:34.415170 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:34.415577 (+ 407us) server_negotiation.cc:407] Connection header received
0810 11:38:34.415808 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:34.415820 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:34.415882 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:34.415997 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:34.417113 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.417974 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.420198 (+ 2224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.420823 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.421020 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.427071 (+ 6051us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:37667: BlockingRecv error: recv got EOF from 127.3.240.195:37667 (error 108)
Metrics: {"server-negotiator.queue_time_us":179}
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:34 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:34.570169 17550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:34.559209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:54221)
0810 11:38:34.560032 (+ 823us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:34.560061 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:34.560343 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:34.560921 (+ 578us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:34.560927 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:34.561712 (+ 785us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:34.561956 (+ 244us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.561966 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.563186 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.563192 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.564145 (+ 953us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:34.564158 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.564933 (+ 775us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.564938 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:34.565069 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.565806 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:34.565826 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:34.569899 (+ 4073us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":321,"threads_started":1}
W20260810 11:38:34.570554 17547 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:34.559380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54221 (local address 127.3.240.254:39665)
0810 11:38:34.559576 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:34.559581 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:34.560300 (+ 719us) server_negotiation.cc:407] Connection header received
0810 11:38:34.560566 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:34.560572 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:34.560690 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:34.560818 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:34.562123 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.562984 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.564324 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:34.564751 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:34.564858 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:34.570112 (+ 5254us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:54221: BlockingRecv error: recv got EOF from 127.3.240.194:54221 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.193@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:35.209975 17551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:35.198079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.193:57079)
0810 11:38:35.198648 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:35.198669 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:35.199060 (+ 391us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:35.199682 (+ 622us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:35.199689 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:35.200431 (+ 742us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:35.200772 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.200789 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.202356 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.202362 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.203548 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.203562 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.204412 (+ 850us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.204417 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.204565 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.205305 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:35.205327 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:35.209665 (+ 4338us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":298,"threads_started":1}
W20260810 11:38:35.210265 17554 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:35.198320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57079 (local address 127.3.240.254:39665)
0810 11:38:35.198980 (+ 660us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:35.198986 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:35.199010 (+ 24us) server_negotiation.cc:407] Connection header received
0810 11:38:35.199236 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:35.199243 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:35.199345 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:35.199443 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:35.201069 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.202121 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.203740 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.204248 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.204375 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.209999 (+ 5624us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.193:57079: BlockingRecv error: recv got EOF from 127.3.240.193:57079 (error 108)
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":385,"threads_started":1}
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.195@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:35.438796 17555 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:35.428604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.195:60323)
0810 11:38:35.429231 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:35.429250 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:35.429431 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:35.429816 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:35.429821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:35.430385 (+ 564us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:35.430599 (+ 214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.430610 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.431872 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.431880 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.432889 (+ 1009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.432901 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.433669 (+ 768us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.433675 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.433869 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.434591 (+ 722us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:35.434611 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:35.438331 (+ 3720us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":356,"threads_started":1}
W20260810 11:38:35.438977 17554 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:35.428679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:60323 (local address 127.3.240.254:39665)
0810 11:38:35.429056 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:35.429064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:35.429362 (+ 298us) server_negotiation.cc:407] Connection header received
0810 11:38:35.429577 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:35.429583 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:35.429660 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:35.429781 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:35.430761 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.431702 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.433059 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.433507 (+ 448us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.433651 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.438563 (+ 4912us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.195:60323: BlockingRecv error: recv got EOF from 127.3.240.195:60323 (error 108)
Metrics: {"server-negotiator.queue_time_us":186}
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
Aug 10 11:38:35 dist-test-slave-drl0 krb5kdc[15926](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.3.240.194@KRBTEST.COM for oryx/127.3.240.254@KRBTEST.COM, Server not found in Kerberos database
I20260810 11:38:35.586758 17556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:35.571528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39665 (local address 127.3.240.194:42143)
0810 11:38:35.572282 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:35.572300 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:35.572494 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:35.572994 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:35.573083 (+ 89us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:35.573505 (+ 422us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:35.573707 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.573717 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.575457 (+ 1740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.575465 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.577250 (+ 1785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:35.577437 (+ 187us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.578990 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.579001 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:35.579298 (+ 297us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.580408 (+ 1110us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:35.580442 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:35.586329 (+ 5887us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.3.240.254:39665: Server oryx/127.3.240.254@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":447,"threads_started":1}
W20260810 11:38:35.587001 17554 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:35.571636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42143 (local address 127.3.240.254:39665)
0810 11:38:35.571941 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:35.571951 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:35.572441 (+ 490us) server_negotiation.cc:407] Connection header received
0810 11:38:35.572647 (+ 206us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:35.572650 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:35.572713 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:35.572809 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:35.573928 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.574902 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.577698 (+ 2796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:35.578630 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:35.578836 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:35.586645 (+ 7809us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.3.240.194:42143: BlockingRecv error: recv got EOF from 127.3.240.194:42143 (error 108)
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 11:38:36.091240 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16837
I20260810 11:38:36.107571 17562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:36.092955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53350 (local address 127.3.240.193:43585)
0810 11:38:36.093859 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:36.093867 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:36.093894 (+ 27us) server_negotiation.cc:407] Connection header received
0810 11:38:36.093958 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:36.093964 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:36.094312 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:36.094517 (+ 205us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:36.096552 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.097677 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.099268 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.099980 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.100095 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:36.103848 (+ 3753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:36.103858 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:36.103872 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:36.103937 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:36.107026 (+ 3089us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 11:38:36.107350 (+ 324us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53350: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":696,"thread_start_us":452,"threads_started":1}
W20260810 11:38:36.107712 17562 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53350: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 11:38:36.107815 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16837: Not authorized: Client connection negotiation failed: client connection to 127.3.240.193:43585: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.193@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 11:38:36.108064 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 16837
I20260810 11:38:36.145130 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17062
I20260810 11:38:36.164192 17563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:36.146689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52514 (local address 127.3.240.194:37653)
0810 11:38:36.147422 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:36.147432 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:36.147464 (+ 32us) server_negotiation.cc:407] Connection header received
0810 11:38:36.147542 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:36.147550 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:36.147884 (+ 334us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:36.148100 (+ 216us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:36.149468 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.150536 (+ 1068us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.152119 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.152907 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.153279 (+ 372us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:36.160225 (+ 6946us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:36.160235 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:36.160250 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:36.160301 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:36.163550 (+ 3249us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 11:38:36.163898 (+ 348us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52514: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":384,"threads_started":1}
W20260810 11:38:36.164320 17563 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:52514: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 11:38:36.164398 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17062: Not authorized: Client connection negotiation failed: client connection to 127.3.240.194:37653: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.194@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 11:38:36.164577 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 17062
I20260810 11:38:36.192334 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17255
I20260810 11:38:36.207377 17566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:36.194141 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36296 (local address 127.3.240.195:33761)
0810 11:38:36.195043 (+ 902us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:36.195055 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:36.195092 (+ 37us) server_negotiation.cc:407] Connection header received
0810 11:38:36.195166 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:36.195173 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:36.195444 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:36.195668 (+ 224us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:36.197141 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.198160 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.199702 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:36.200322 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:36.200483 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:36.203441 (+ 2958us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:36.203451 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:36.203466 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:36.203506 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:36.206459 (+ 2953us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
0810 11:38:36.206960 (+ 501us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36296: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":443,"threads_started":1}
W20260810 11:38:36.207346 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17255: Not authorized: Client connection negotiation failed: client connection to 127.3.240.195:33761: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260810 11:38:36.207489 17566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36296: Request ticket server kudu/127.3.240.195@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260810 11:38:36.207552 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 17255
I20260810 11:38:36.236783 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 16739
I20260810 11:38:36.238396 16829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:36.369238 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 16739
2026-08-10T11:38:36Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22821 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-10T11:38:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:36Z Disabled control of system clock
I20260810 11:38:36.487677 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35697
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35697
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260810 11:38:36.897436 17574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:36.897737 17574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:36.897861 17574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:36.905155 17574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:36.905261 17574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:36.905339 17574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:36.905428 17574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:36.905494 17574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:36.914808 17574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35697
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:36.916203 17574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:36.918032 17574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:36.931154 17582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:36.931210 17584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:36.931977 17574 server_base.cc:1034] running on GCE node
W20260810 11:38:36.931954 17588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:36.933238 17574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:36.935181 17574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:36.936595 17574 hybrid_clock.cc:648] HybridClock initialized: now 1786361916936522 us; error 44 us; skew 500 ppm
I20260810 11:38:36.937103 17574 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:36.940619 17574 webserver.cc:460] Webserver started at http://127.3.240.254:46663/ using document root <none> and password file <none>
I20260810 11:38:36.941619 17574 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:36.941733 17574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:36.942174 17574 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:36.945407 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "94f14e927c08416ab09a29a6f0e10338"
format_stamp: "Formatted at 2026-08-10 11:38:36 on dist-test-slave-drl0"
server_key: "9436258528e3118d151291bf91a555dd"
server_key_iv: "9b5d16239e1a2511700c837349ac307c"
server_key_version: "encryptionkey@0"
I20260810 11:38:36.946200 17574 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "94f14e927c08416ab09a29a6f0e10338"
format_stamp: "Formatted at 2026-08-10 11:38:36 on dist-test-slave-drl0"
server_key: "9436258528e3118d151291bf91a555dd"
server_key_iv: "9b5d16239e1a2511700c837349ac307c"
server_key_version: "encryptionkey@0"
I20260810 11:38:36.951853 17574 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 11:38:36.956091 17596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:36.957705 17574 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:38:36.957926 17574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "94f14e927c08416ab09a29a6f0e10338"
format_stamp: "Formatted at 2026-08-10 11:38:36 on dist-test-slave-drl0"
server_key: "9436258528e3118d151291bf91a555dd"
server_key_iv: "9b5d16239e1a2511700c837349ac307c"
server_key_version: "encryptionkey@0"
I20260810 11:38:36.958160 17574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:36.987126 17574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:36.988001 17574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:36.988322 17574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:37.011961 17574 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35697
I20260810 11:38:37.012007 17680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35697 every 8 connection(s)
I20260810 11:38:37.013381 17574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:37.014214 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 17574
I20260810 11:38:37.014441 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:38:37.015290 4035 external_mini_cluster.cc:1428] Setting key be1c0faf02c93ba73f38bb95bb8f7ff7
I20260810 11:38:37.019387 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:37.031116 17685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:37.017548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37392 (local address 127.3.240.254:35697)
0810 11:38:37.018481 (+ 933us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:37.018492 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:37.018536 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:38:37.020342 (+ 1806us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:37.020362 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:37.020789 (+ 427us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:37.021112 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:37.021960 (+ 848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.023585 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.025543 (+ 1958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.026178 (+ 635us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.026315 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:37.028062 (+ 1747us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:37.028079 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:37.028101 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:37.029010 (+ 909us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:37.029226 (+ 216us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:37.029279 (+ 53us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:37.029386 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:37.029780 (+ 394us) server_negotiation.cc:299] Negotiation successful
0810 11:38:37.030101 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":307,"threads_started":1}
I20260810 11:38:37.032101 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338: Bootstrap starting.
I20260810 11:38:37.035411 17681 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:37.036470 17681 log.cc:824] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:37.039992 17681 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338: No bootstrap required, opened a new log
I20260810 11:38:37.047818 17681 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } }
I20260810 11:38:37.048179 17681 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:37.048287 17681 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94f14e927c08416ab09a29a6f0e10338, State: Initialized, Role: FOLLOWER
I20260810 11:38:37.048895 17681 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } }
I20260810 11:38:37.049098 17681 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:37.049253 17681 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:37.049437 17681 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:37.051860 17681 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } }
I20260810 11:38:37.052470 17681 leader_election.cc:302] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94f14e927c08416ab09a29a6f0e10338; no voters:
I20260810 11:38:37.053316 17681 leader_election.cc:288] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:38:37.053489 17688 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:37.054230 17688 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [term 1 LEADER]: Becoming Leader. State: Replica: 94f14e927c08416ab09a29a6f0e10338, State: Running, Role: LEADER
I20260810 11:38:37.054827 17688 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } }
I20260810 11:38:37.055486 17681 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:37.057916 17692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94f14e927c08416ab09a29a6f0e10338. Latest consensus state: current_term: 1 leader_uuid: "94f14e927c08416ab09a29a6f0e10338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } } }
I20260810 11:38:37.058372 17692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:37.057966 17689 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94f14e927c08416ab09a29a6f0e10338" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94f14e927c08416ab09a29a6f0e10338" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35697 } } }
I20260810 11:38:37.059173 17689 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338 [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:37.062680 17700 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:37.069597 17700 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:37.079723 17700 catalog_manager.cc:1347] Generated new cluster ID: d2b8d3c362b1456f98fcc98478261ff9
I20260810 11:38:37.079856 17700 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:37.097851 17700 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:38:37.099761 17700 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:37.123953 17700 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94f14e927c08416ab09a29a6f0e10338: Generated new TSK 0
I20260810 11:38:37.125150 17700 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:38:37.132115 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--builtin_ntp_servers=127.3.240.212:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260810 11:38:37.542850 17718 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:37.543262 17718 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:37.543426 17718 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:37.550995 17718 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:37.551143 17718 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:37.551324 17718 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:37.563788 17718 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:37.565295 17718 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:37.567221 17718 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:37.583041 17725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:37.582953 17729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:37.582986 17727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:37.583706 17718 server_base.cc:1034] running on GCE node
I20260810 11:38:37.584371 17718 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:37.585350 17718 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:37.586601 17718 hybrid_clock.cc:648] HybridClock initialized: now 1786361917586580 us; error 46 us; skew 500 ppm
I20260810 11:38:37.587101 17718 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:37.589633 17718 webserver.cc:460] Webserver started at http://127.3.240.193:36289/ using document root <none> and password file <none>
I20260810 11:38:37.590433 17718 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:37.590585 17718 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:37.591010 17718 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:37.594120 17718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "d85de78585864f9aad5d716acb3b38f3"
format_stamp: "Formatted at 2026-08-10 11:38:37 on dist-test-slave-drl0"
server_key: "4b52d2e83561c8d44a7d284550e89c67"
server_key_iv: "f33291e709236a6575f7a3bedc8da2dd"
server_key_version: "encryptionkey@0"
I20260810 11:38:37.595027 17718 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "d85de78585864f9aad5d716acb3b38f3"
format_stamp: "Formatted at 2026-08-10 11:38:37 on dist-test-slave-drl0"
server_key: "4b52d2e83561c8d44a7d284550e89c67"
server_key_iv: "f33291e709236a6575f7a3bedc8da2dd"
server_key_version: "encryptionkey@0"
I20260810 11:38:37.600474 17718 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 11:38:37.605305 17740 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:37.606802 17718 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.001s
I20260810 11:38:37.607123 17718 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "d85de78585864f9aad5d716acb3b38f3"
format_stamp: "Formatted at 2026-08-10 11:38:37 on dist-test-slave-drl0"
server_key: "4b52d2e83561c8d44a7d284550e89c67"
server_key_iv: "f33291e709236a6575f7a3bedc8da2dd"
server_key_version: "encryptionkey@0"
I20260810 11:38:37.607362 17718 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:37.642274 17718 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:37.643751 17718 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:37.644157 17718 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:37.645255 17718 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:37.647042 17718 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:37.647161 17718 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:37.647301 17718 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:37.647393 17718 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:37.689714 17718 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:45759
I20260810 11:38:37.689818 17903 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:45759 every 8 connection(s)
I20260810 11:38:37.691105 17718 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:37.694591 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 17718
I20260810 11:38:37.694804 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:38:37.695577 4035 external_mini_cluster.cc:1428] Setting key 6178f8c21f4be2fe6057026f7ac2b64d
I20260810 11:38:37.699195 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--builtin_ntp_servers=127.3.240.212:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 11:38:37.707129 17907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:37.694265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49321 (local address 127.3.240.254:35697)
0810 11:38:37.695019 (+ 754us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:37.695027 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:37.696589 (+ 1562us) server_negotiation.cc:407] Connection header received
0810 11:38:37.698538 (+ 1949us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:37.698543 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:37.698624 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:37.698761 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:37.700421 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.701217 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.703710 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.704126 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.704268 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:37.705957 (+ 1689us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:37.705963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:37.705966 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:37.705987 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:37.706026 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:37.706038 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:37.706110 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:37.706447 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 11:38:37.706533 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":350,"threads_started":1}
I20260810 11:38:37.708331 17910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:37.695026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35697 (local address 127.3.240.193:49321)
0810 11:38:37.696387 (+ 1361us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:37.696437 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:37.698168 (+ 1731us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:37.699087 (+ 919us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:37.699098 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:37.699213 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:37.700190 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:37.700220 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.701384 (+ 1164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.701392 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:37.703505 (+ 2113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:37.703524 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:37.704383 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:37.704393 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:37.704721 (+ 328us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:37.705540 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:37.705546 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:37.705601 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:37.705701 (+ 100us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:37.706137 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:37.706150 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:37.706161 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 11:38:37.706347 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 11:38:37.706755 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":684,"thread_start_us":420,"threads_started":1}
I20260810 11:38:37.710147 17904 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35697
I20260810 11:38:37.710513 17904 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:37.711962 17904 heartbeater.cc:511] Master 127.3.240.254:35697 requested a full tablet report, sending...
I20260810 11:38:37.714906 17620 ts_manager.cc:194] Registered new tserver with Master: d85de78585864f9aad5d716acb3b38f3 (127.3.240.193:45759)
I20260810 11:38:37.716679 17620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:49321
W20260810 11:38:38.081993 17911 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:38.082422 17911 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:38.082628 17911 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:38.091056 17911 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:38.091182 17911 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:38.091374 17911 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:38.101229 17911 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:38.102593 17911 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:38.104786 17911 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:38.120507 17916 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:38.120544 17919 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:38.120709 17917 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:38.121474 17911 server_base.cc:1034] running on GCE node
I20260810 11:38:38.122305 17911 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:38.123481 17911 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:38.124751 17911 hybrid_clock.cc:648] HybridClock initialized: now 1786361918124697 us; error 44 us; skew 500 ppm
I20260810 11:38:38.125135 17911 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:38.127919 17911 webserver.cc:460] Webserver started at http://127.3.240.194:41683/ using document root <none> and password file <none>
I20260810 11:38:38.128684 17911 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:38.128785 17911 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:38.129156 17911 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:38.132241 17911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "45217cb58fad4a77ac9e0918a13f809e"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "7159f11c3d59b75471e8522e411dd1a9"
server_key_iv: "9f61ec4700424fd46affbdfd863021db"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.133054 17911 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "45217cb58fad4a77ac9e0918a13f809e"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "7159f11c3d59b75471e8522e411dd1a9"
server_key_iv: "9f61ec4700424fd46affbdfd863021db"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.138756 17911 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 11:38:38.143573 17930 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.145272 17911 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:38:38.145521 17911 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "45217cb58fad4a77ac9e0918a13f809e"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "7159f11c3d59b75471e8522e411dd1a9"
server_key_iv: "9f61ec4700424fd46affbdfd863021db"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.145803 17911 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:38.180135 17911 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:38.181385 17911 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:38.181774 17911 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:38.183071 17911 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:38.185042 17911 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:38.185199 17911 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.185359 17911 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:38.185463 17911 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.229560 17911 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36635
I20260810 11:38:38.229650 18096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36635 every 8 connection(s)
I20260810 11:38:38.231057 17911 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:38.239063 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 17911
I20260810 11:38:38.239244 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:38:38.239742 4035 external_mini_cluster.cc:1428] Setting key 5b73db3617739d7e5bc278046b37fb83
I20260810 11:38:38.243140 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--builtin_ntp_servers=127.3.240.212:39567
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260810 11:38:38.246572 18101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.234611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:37695 (local address 127.3.240.254:35697)
0810 11:38:38.235173 (+ 562us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:38.235178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:38.236507 (+ 1329us) server_negotiation.cc:407] Connection header received
0810 11:38:38.238521 (+ 2014us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:38.238528 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:38.238614 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:38.238736 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:38.239968 (+ 1232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.240845 (+ 877us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.242648 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.243484 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.243668 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.245353 (+ 1685us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:38.245363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:38.245372 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:38.245403 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:38.245468 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:38.245492 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:38.245617 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:38.245890 (+ 273us) server_negotiation.cc:299] Negotiation successful
0810 11:38:38.246019 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":441,"thread_start_us":276,"threads_started":1}
I20260810 11:38:38.247831 18102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.235243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35697 (local address 127.3.240.194:37695)
0810 11:38:38.236330 (+ 1087us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:38.236373 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:38.238199 (+ 1826us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:38.239026 (+ 827us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:38.239033 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:38.239114 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:38.239750 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:38.239766 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.241045 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.241049 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:38.242467 (+ 1418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:38.242481 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.243813 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.243818 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:38.244150 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.245009 (+ 859us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:38.245012 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:38.245052 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:38.245126 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:38.245641 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:38.245651 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:38.245659 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 11:38:38.245835 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 11:38:38.246149 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":578,"thread_start_us":328,"threads_started":1}
I20260810 11:38:38.249792 18097 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35697
I20260810 11:38:38.250130 18097 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:38.251073 18097 heartbeater.cc:511] Master 127.3.240.254:35697 requested a full tablet report, sending...
I20260810 11:38:38.252732 17620 ts_manager.cc:194] Registered new tserver with Master: 45217cb58fad4a77ac9e0918a13f809e (127.3.240.194:36635)
I20260810 11:38:38.253558 17620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:37695
W20260810 11:38:38.617908 18104 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:38.618259 18104 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:38.618520 18104 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:38.627054 18104 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:38.627158 18104 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:38.627272 18104 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:38.636960 18104 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39567
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35697
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:38.638397 18104 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:38.640189 18104 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:38.657532 18114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:38.657542 18111 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:38.657599 18120 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:38.658345 18104 server_base.cc:1034] running on GCE node
I20260810 11:38:38.659844 18104 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:38.661130 18104 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:38.662374 18104 hybrid_clock.cc:648] HybridClock initialized: now 1786361918662327 us; error 63 us; skew 500 ppm
I20260810 11:38:38.662835 18104 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:38.665922 18104 webserver.cc:460] Webserver started at http://127.3.240.195:39393/ using document root <none> and password file <none>
I20260810 11:38:38.667127 18104 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:38.667320 18104 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:38.667943 18104 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:38.673681 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "79959cc1f37844f9adbaf08d3d6e79fc"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "db50c3fd9e73dbeaaf3c58019efbaa66"
server_key_iv: "2c230aa86a7978c3f8c6e5a4fa76a279"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.674563 18104 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "79959cc1f37844f9adbaf08d3d6e79fc"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "db50c3fd9e73dbeaaf3c58019efbaa66"
server_key_iv: "2c230aa86a7978c3f8c6e5a4fa76a279"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.680429 18104 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260810 11:38:38.684273 18131 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.685678 18104 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 11:38:38.685883 18104 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "79959cc1f37844f9adbaf08d3d6e79fc"
format_stamp: "Formatted at 2026-08-10 11:38:38 on dist-test-slave-drl0"
server_key: "db50c3fd9e73dbeaaf3c58019efbaa66"
server_key_iv: "2c230aa86a7978c3f8c6e5a4fa76a279"
server_key_version: "encryptionkey@0"
I20260810 11:38:38.686126 18104 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:38.700165 18104 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:38.701043 18104 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:38.701349 18104 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:38.702315 18104 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:38.704075 18104 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:38.704314 18104 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.704587 18104 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:38.704707 18104 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:38.721333 17904 heartbeater.cc:503] Master 127.3.240.254:35697 was elected leader, sending a full tablet report...
I20260810 11:38:38.747694 18104 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:38157
I20260810 11:38:38.747757 18306 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:38157 every 8 connection(s)
I20260810 11:38:38.749612 18104 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:38.750012 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 18104
I20260810 11:38:38.750190 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:38:38.750885 4035 external_mini_cluster.cc:1428] Setting key f17ae9d7b459f1c08516722bb4d1804c
I20260810 11:38:38.765614 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.752609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:35077 (local address 127.3.240.254:35697)
0810 11:38:38.755323 (+ 2714us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:38.755328 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:38.755344 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:38:38.757052 (+ 1708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:38.757056 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:38.757209 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:38.757451 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:38.759067 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.759863 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.761562 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.762335 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.762507 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.764541 (+ 2034us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:38.764546 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:38.764550 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:38.764568 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:38.764607 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:38.764619 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:38.764712 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:38.765067 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 11:38:38.765210 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2614,"thread_start_us":2484,"threads_started":1}
I20260810 11:38:38.766685 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.753465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35697 (local address 127.3.240.195:35077)
0810 11:38:38.754708 (+ 1243us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:38.754755 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:38.756754 (+ 1999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:38.757860 (+ 1106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:38.757873 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:38.758002 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:38.758805 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:38.758828 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.760028 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.760032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:38.761373 (+ 1341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:38.761387 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.762762 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.762771 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:38.763205 (+ 434us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.764195 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:38.764198 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:38.764241 (+ 43us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:38.764321 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:38.764801 (+ 480us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:38.764814 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:38.764823 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 11:38:38.765036 (+ 213us) client_negotiation.cc:240] Negotiation successful
0810 11:38:38.765414 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":388,"threads_started":1}
I20260810 11:38:38.768184 18307 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35697
I20260810 11:38:38.768467 18307 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:38.769186 18307 heartbeater.cc:511] Master 127.3.240.254:35697 requested a full tablet report, sending...
I20260810 11:38:38.770556 17620 ts_manager.cc:194] Registered new tserver with Master: 79959cc1f37844f9adbaf08d3d6e79fc (127.3.240.195:38157)
I20260810 11:38:38.771442 17620 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:35077
I20260810 11:38:38.780486 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 11:38:38.787155 18310 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:38.785371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37398 (local address 127.3.240.254:35697)
0810 11:38:38.785562 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:38.785567 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:38.786136 (+ 569us) server_negotiation.cc:407] Connection header received
0810 11:38:38.786249 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:38.786253 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:38.786314 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:38.786403 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:38.786884 (+ 481us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37398: BlockingRecv error: recv got EOF from 127.0.0.1:37398 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 11:38:38.788128 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17718
I20260810 11:38:38.798290 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.789159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49042 (local address 127.3.240.193:45759)
0810 11:38:38.789738 (+ 579us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:38.789774 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:38.789800 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:38:38.789858 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:38.789864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:38.790078 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:38.790235 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:38.791140 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.792336 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.793869 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.794457 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.794567 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.796185 (+ 1618us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:38.796199 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:38.796221 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:38.797044 (+ 823us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:38.797118 (+ 74us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:38.797145 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:38.797276 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:38.797623 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 11:38:38.797806 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":274,"threads_started":1}
I20260810 11:38:38.799963 17896 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:38.890240 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 17718
I20260810 11:38:38.916344 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17911
I20260810 11:38:38.926359 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:38.917605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.3.240.194:36635)
0810 11:38:38.918210 (+ 605us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:38.918218 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:38.918264 (+ 46us) server_negotiation.cc:407] Connection header received
0810 11:38:38.918347 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:38.918353 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:38.918589 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:38.918721 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:38.919663 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.920838 (+ 1175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.922265 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:38.923100 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:38.923243 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:38.924655 (+ 1412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:38.924667 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:38.924682 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:38.925346 (+ 664us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:38.925405 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:38.925426 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:38.925564 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:38.925796 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 11:38:38.925999 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":277,"threads_started":1}
I20260810 11:38:38.928125 18091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:39.023288 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 17911
I20260810 11:38:39.055632 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 18104
I20260810 11:38:39.066496 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:39.056909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48232 (local address 127.3.240.195:38157)
0810 11:38:39.057591 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:39.057597 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:39.057627 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:38:39.057699 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:39.057705 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:39.057926 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:39.058076 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:39.058614 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:39.059820 (+ 1206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:39.061564 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:39.062758 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:39.062872 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:39.064483 (+ 1611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:39.064498 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:39.064520 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:39.065357 (+ 837us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:39.065429 (+ 72us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:39.065462 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:39.065617 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:39.065861 (+ 244us) server_negotiation.cc:299] Negotiation successful
0810 11:38:39.066097 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":376,"threads_started":1}
I20260810 11:38:39.068331 18298 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:39.158623 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 18104
I20260810 11:38:39.183966 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 17574
I20260810 11:38:39.185340 17669 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:39.295989 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 17574
2026-08-10T11:38:39Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2915 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-10T11:38:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:39Z Disabled control of system clock
I20260810 11:38:39.395514 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38073
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:38073
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 11:38:39.770648 18331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:39.771049 18331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:39.771225 18331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:39.778843 18331 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:39.778996 18331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:39.779076 18331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:39.779168 18331 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:39.779235 18331 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:39.790364 18331 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:38073
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:38073
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:39.791832 18331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:39.793864 18331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:39.807932 18341 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:39.808229 18343 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:39.808629 18345 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:39.809141 18331 server_base.cc:1034] running on GCE node
I20260810 11:38:39.810259 18331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:39.812175 18331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:39.813442 18331 hybrid_clock.cc:648] HybridClock initialized: now 1786361919813356 us; error 65 us; skew 500 ppm
I20260810 11:38:39.813948 18331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:39.817402 18331 webserver.cc:460] Webserver started at http://127.3.240.254:37985/ using document root <none> and password file <none>
I20260810 11:38:39.818416 18331 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:39.818526 18331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:39.818990 18331 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:39.822667 18331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "edd910de6afe4ce2ac72980cb983982f"
format_stamp: "Formatted at 2026-08-10 11:38:39 on dist-test-slave-drl0"
server_key: "3d0fade81b78919a56b90aef3440ad6e"
server_key_iv: "0c73afa625d9617495c0977dcd64ca41"
server_key_version: "encryptionkey@0"
I20260810 11:38:39.823846 18331 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "edd910de6afe4ce2ac72980cb983982f"
format_stamp: "Formatted at 2026-08-10 11:38:39 on dist-test-slave-drl0"
server_key: "3d0fade81b78919a56b90aef3440ad6e"
server_key_iv: "0c73afa625d9617495c0977dcd64ca41"
server_key_version: "encryptionkey@0"
I20260810 11:38:39.829686 18331 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:38:39.834105 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:39.836098 18331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:38:39.836277 18331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "edd910de6afe4ce2ac72980cb983982f"
format_stamp: "Formatted at 2026-08-10 11:38:39 on dist-test-slave-drl0"
server_key: "3d0fade81b78919a56b90aef3440ad6e"
server_key_iv: "0c73afa625d9617495c0977dcd64ca41"
server_key_version: "encryptionkey@0"
I20260810 11:38:39.836560 18331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:39.844812 18331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:39.845927 18331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:39.846339 18331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:39.870811 18331 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:38073
I20260810 11:38:39.870832 18438 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:38073 every 8 connection(s)
I20260810 11:38:39.872440 18331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:39.876782 18439 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:39.879927 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 18331
I20260810 11:38:39.880164 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:38:39.881052 4035 external_mini_cluster.cc:1428] Setting key 172587c23152bbb07c9320c51e6a8744
I20260810 11:38:39.890486 18455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:39.883420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58264 (local address 127.3.240.254:38073)
0810 11:38:39.884480 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:39.884491 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:39.884537 (+ 46us) server_negotiation.cc:407] Connection header received
0810 11:38:39.886548 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:39.886574 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:39.887102 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:39.887458 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:39.888596 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:39.888604 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:39.888619 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:39.888707 (+ 88us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:39.888793 (+ 86us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:39.888822 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:39.889044 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:39.889149 (+ 105us) server_negotiation.cc:299] Negotiation successful
0810 11:38:39.889532 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":643,"thread_start_us":401,"threads_started":1}
I20260810 11:38:39.890799 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f: Bootstrap starting.
I20260810 11:38:39.895222 18439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:39.896589 18439 log.cc:824] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:39.900574 18439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f: No bootstrap required, opened a new log
I20260810 11:38:39.909751 18439 raft_consensus.cc:348] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } }
I20260810 11:38:39.910342 18439 raft_consensus.cc:374] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:39.910475 18439 raft_consensus.cc:729] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd910de6afe4ce2ac72980cb983982f, State: Initialized, Role: FOLLOWER
I20260810 11:38:39.911360 18439 consensus_queue.cc:260] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } }
I20260810 11:38:39.911630 18439 raft_consensus.cc:388] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:39.911751 18439 raft_consensus.cc:482] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:39.911900 18439 raft_consensus.cc:3037] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:39.914392 18439 raft_consensus.cc:504] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } }
I20260810 11:38:39.915060 18439 leader_election.cc:302] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: edd910de6afe4ce2ac72980cb983982f; no voters:
I20260810 11:38:39.915994 18439 leader_election.cc:288] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:38:39.916297 18457 raft_consensus.cc:2781] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:39.917551 18457 raft_consensus.cc:686] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [term 1 LEADER]: Becoming Leader. State: Replica: edd910de6afe4ce2ac72980cb983982f, State: Running, Role: LEADER
I20260810 11:38:39.918560 18439 sys_catalog.cc:564] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:39.918498 18457 consensus_queue.cc:237] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } }
I20260810 11:38:39.922518 18458 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "edd910de6afe4ce2ac72980cb983982f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } } }
I20260810 11:38:39.922487 18460 sys_catalog.cc:455] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [sys.catalog]: SysCatalogTable state changed. Reason: New leader edd910de6afe4ce2ac72980cb983982f. Latest consensus state: current_term: 1 leader_uuid: "edd910de6afe4ce2ac72980cb983982f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "edd910de6afe4ce2ac72980cb983982f" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 38073 } } }
I20260810 11:38:39.923151 18460 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:39.923152 18458 sys_catalog.cc:458] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:39.924396 18471 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:39.931124 18471 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:39.942257 18471 catalog_manager.cc:1347] Generated new cluster ID: d878bb430e8046b2a4d1a2efc2f2919c
I20260810 11:38:39.942384 18471 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:39.964171 18471 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:38:39.964795 18471 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:39.971432 18471 catalog_manager.cc:5868] T 00000000000000000000000000000000 P edd910de6afe4ce2ac72980cb983982f: Generated new TSK 0
I20260810 11:38:39.972307 18471 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:38:39.978734 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--builtin_ntp_servers=127.3.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260810 11:38:40.354816 18490 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:40.355191 18490 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:40.355321 18490 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:40.362515 18490 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:40.362615 18490 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:40.362776 18490 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:40.373145 18490 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:40.375321 18490 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:40.378144 18490 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:40.394721 18499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:40.394672 18501 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:40.394739 18504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:40.396066 18490 server_base.cc:1034] running on GCE node
I20260810 11:38:40.396714 18490 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:40.397811 18490 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:40.399101 18490 hybrid_clock.cc:648] HybridClock initialized: now 1786361920399015 us; error 79 us; skew 500 ppm
I20260810 11:38:40.399467 18490 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:40.402519 18490 webserver.cc:460] Webserver started at http://127.3.240.193:34823/ using document root <none> and password file <none>
I20260810 11:38:40.403637 18490 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:40.403745 18490 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:40.404172 18490 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:40.408010 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "8f4ee1e74a414f80a5a0ff910e366ada"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "d6670320d395a83a3bb89f8cb1d54b1c"
server_key_iv: "428c7e4e8d0c6e0087a9366b35f90d49"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.408933 18490 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "8f4ee1e74a414f80a5a0ff910e366ada"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "d6670320d395a83a3bb89f8cb1d54b1c"
server_key_iv: "428c7e4e8d0c6e0087a9366b35f90d49"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.415179 18490 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:38:40.418851 18513 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.420439 18490 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:38:40.420621 18490 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "8f4ee1e74a414f80a5a0ff910e366ada"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "d6670320d395a83a3bb89f8cb1d54b1c"
server_key_iv: "428c7e4e8d0c6e0087a9366b35f90d49"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.421001 18490 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:40.428791 18490 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:40.429663 18490 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:40.429957 18490 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:40.431000 18490 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:40.432510 18490 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:40.432600 18490 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.432742 18490 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:40.432826 18490 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.474153 18490 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37411
I20260810 11:38:40.474241 18673 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37411 every 8 connection(s)
I20260810 11:38:40.475584 18490 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:40.484568 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 18490
I20260810 11:38:40.484817 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:38:40.485827 4035 external_mini_cluster.cc:1428] Setting key fc4d290af9bf82101192b5a69bff6136
I20260810 11:38:40.486354 18676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:40.478313 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:51657 (local address 127.3.240.254:38073)
0810 11:38:40.478843 (+ 530us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:40.478849 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:40.480437 (+ 1588us) server_negotiation.cc:407] Connection header received
0810 11:38:40.483338 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:40.483343 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:40.483399 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:40.483510 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:40.485069 (+ 1559us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:40.485074 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:40.485077 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:40.485094 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:40.485147 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:40.485160 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:40.485269 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:40.485741 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 11:38:40.485844 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":232,"threads_started":1}
I20260810 11:38:40.487198 18677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:40.479051 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38073 (local address 127.3.240.193:51657)
0810 11:38:40.480234 (+ 1183us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:40.480290 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:40.483020 (+ 2730us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:40.483812 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:40.483820 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:40.483912 (+ 92us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:40.484698 (+ 786us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:40.484702 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:40.484759 (+ 57us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:40.484881 (+ 122us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:40.485292 (+ 411us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:40.485306 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:40.485315 (+ 9us) client_negotiation.cc:769] Sending connection context
0810 11:38:40.485648 (+ 333us) client_negotiation.cc:240] Negotiation successful
0810 11:38:40.485949 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":346,"threads_started":1}
I20260810 11:38:40.489355 18674 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38073
I20260810 11:38:40.489629 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--builtin_ntp_servers=127.3.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 11:38:40.489717 18674 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:40.490445 18674 heartbeater.cc:511] Master 127.3.240.254:38073 requested a full tablet report, sending...
I20260810 11:38:40.492980 18381 ts_manager.cc:194] Registered new tserver with Master: 8f4ee1e74a414f80a5a0ff910e366ada (127.3.240.193:37411)
W20260810 11:38:40.867146 18678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:40.867578 18678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:40.867690 18678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:40.879664 18678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:40.879812 18678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:40.880009 18678 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:40.891995 18678 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:40.893281 18678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:40.895144 18678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:40.909369 18684 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:40.909466 18685 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:40.910089 18678 server_base.cc:1034] running on GCE node
W20260810 11:38:40.910166 18687 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:40.910825 18678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:40.911844 18678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:40.913196 18678 hybrid_clock.cc:648] HybridClock initialized: now 1786361920913123 us; error 30 us; skew 500 ppm
I20260810 11:38:40.913769 18678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:40.916965 18678 webserver.cc:460] Webserver started at http://127.3.240.194:40801/ using document root <none> and password file <none>
I20260810 11:38:40.918119 18678 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:40.918304 18678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:40.918756 18678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:40.922698 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "8a316cce18a34e3f8d00a1e0b5a5a3d1"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "cc9d0ef8779871a32018c1547ae31a05"
server_key_iv: "06371a6f7b6d2ab03ce2cf473938e805"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.923965 18678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "8a316cce18a34e3f8d00a1e0b5a5a3d1"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "cc9d0ef8779871a32018c1547ae31a05"
server_key_iv: "06371a6f7b6d2ab03ce2cf473938e805"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.936056 18678 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.011s sys 0.000s
I20260810 11:38:40.940812 18695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.942653 18678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:38:40.942867 18678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "8a316cce18a34e3f8d00a1e0b5a5a3d1"
format_stamp: "Formatted at 2026-08-10 11:38:40 on dist-test-slave-drl0"
server_key: "cc9d0ef8779871a32018c1547ae31a05"
server_key_iv: "06371a6f7b6d2ab03ce2cf473938e805"
server_key_version: "encryptionkey@0"
I20260810 11:38:40.943203 18678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:40.949158 18678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:40.950027 18678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:40.950325 18678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:40.951364 18678 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:40.953713 18678 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:40.953894 18678 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.954069 18678 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:40.954173 18678 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:40.996397 18678 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:40627
I20260810 11:38:40.996490 18856 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:40627 every 8 connection(s)
I20260810 11:38:40.997813 18678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:41.006410 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 18678
I20260810 11:38:41.006701 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:38:41.007323 18864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.000584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:46991 (local address 127.3.240.254:38073)
0810 11:38:41.001101 (+ 517us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.001105 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.002225 (+ 1120us) server_negotiation.cc:407] Connection header received
0810 11:38:41.004259 (+ 2034us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.004265 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.004346 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.004473 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.006282 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:41.006289 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:41.006294 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:41.006319 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:41.006384 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:41.006399 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:41.006612 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:41.006973 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 11:38:41.007057 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":405,"thread_start_us":279,"threads_started":1}
I20260810 11:38:41.007498 4035 external_mini_cluster.cc:1428] Setting key e6b724d25db25b890a32eb7e50c9302f
I20260810 11:38:41.008988 18869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.001108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38073 (local address 127.3.240.194:46991)
0810 11:38:41.002063 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:41.002101 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:41.003972 (+ 1871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:41.004707 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:41.004715 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:41.004803 (+ 88us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:41.005622 (+ 819us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:41.005626 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:41.005692 (+ 66us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:41.005932 (+ 240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:41.006554 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:41.006569 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:41.006583 (+ 14us) client_negotiation.cc:769] Sending connection context
0810 11:38:41.007029 (+ 446us) client_negotiation.cc:240] Negotiation successful
0810 11:38:41.007430 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":302,"threads_started":1}
I20260810 11:38:41.010877 18861 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38073
I20260810 11:38:41.011264 18861 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:41.011873 18861 heartbeater.cc:511] Master 127.3.240.254:38073 requested a full tablet report, sending...
I20260810 11:38:41.011866 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--builtin_ntp_servers=127.3.240.212:44269
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260810 11:38:41.013585 18381 ts_manager.cc:194] Registered new tserver with Master: 8a316cce18a34e3f8d00a1e0b5a5a3d1 (127.3.240.194:40627)
W20260810 11:38:41.400771 18870 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:41.401391 18870 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:41.401777 18870 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:41.410737 18870 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:41.410857 18870 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:41.411087 18870 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:41.420903 18870 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44269
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:38073
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:41.422849 18870 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:41.425365 18870 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:41.442301 18876 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:41.442261 18877 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:41.442251 18880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:41.443360 18870 server_base.cc:1034] running on GCE node
I20260810 11:38:41.444584 18870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:41.445611 18870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:41.446852 18870 hybrid_clock.cc:648] HybridClock initialized: now 1786361921446796 us; error 50 us; skew 500 ppm
I20260810 11:38:41.447341 18870 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:38:41.453114 18870 webserver.cc:460] Webserver started at http://127.3.240.195:46009/ using document root <none> and password file <none>
I20260810 11:38:41.453855 18870 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:41.453944 18870 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:41.454237 18870 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:41.457330 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "b6c837f3ff384531bf4fd974dd5f92be"
format_stamp: "Formatted at 2026-08-10 11:38:41 on dist-test-slave-drl0"
server_key: "9f7c010896c82b8f194ab71019c23071"
server_key_iv: "a7d7cc43493fbc987f3c49012ec4f280"
server_key_version: "encryptionkey@0"
I20260810 11:38:41.458112 18870 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "b6c837f3ff384531bf4fd974dd5f92be"
format_stamp: "Formatted at 2026-08-10 11:38:41 on dist-test-slave-drl0"
server_key: "9f7c010896c82b8f194ab71019c23071"
server_key_iv: "a7d7cc43493fbc987f3c49012ec4f280"
server_key_version: "encryptionkey@0"
I20260810 11:38:41.464079 18870 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:38:41.467672 18887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:41.469074 18870 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 11:38:41.469278 18870 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "b6c837f3ff384531bf4fd974dd5f92be"
format_stamp: "Formatted at 2026-08-10 11:38:41 on dist-test-slave-drl0"
server_key: "9f7c010896c82b8f194ab71019c23071"
server_key_iv: "a7d7cc43493fbc987f3c49012ec4f280"
server_key_version: "encryptionkey@0"
I20260810 11:38:41.469498 18870 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:41.476866 18870 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:41.477720 18870 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:41.478057 18870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:41.479056 18870 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:41.480621 18870 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:41.480748 18870 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:41.480885 18870 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:41.481002 18870 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:41.497201 18674 heartbeater.cc:503] Master 127.3.240.254:38073 was elected leader, sending a full tablet report...
I20260810 11:38:41.528038 18870 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42121
I20260810 11:38:41.528090 19062 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42121 every 8 connection(s)
I20260810 11:38:41.530309 18870 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:41.539331 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 18870
I20260810 11:38:41.539562 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:38:41.540093 4035 external_mini_cluster.cc:1428] Setting key b5562b22bce201a533609d3a33e81a5b
I20260810 11:38:41.540107 19066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.533586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:33485 (local address 127.3.240.254:38073)
0810 11:38:41.534080 (+ 494us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.534085 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.535931 (+ 1846us) server_negotiation.cc:407] Connection header received
0810 11:38:41.537637 (+ 1706us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.537642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.537719 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.537842 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.539269 (+ 1427us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:41.539276 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:41.539281 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:41.539299 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:41.539357 (+ 58us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:41.539371 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:41.539505 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:41.539700 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 11:38:41.539773 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":379,"thread_start_us":257,"threads_started":1}
I20260810 11:38:41.541298 19067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.534122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:38073 (local address 127.3.240.195:33485)
0810 11:38:41.535751 (+ 1629us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:41.535794 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:41.537370 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:41.538079 (+ 709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:41.538087 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:41.538172 (+ 85us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:41.538902 (+ 730us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:38:41.538904 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:41.539012 (+ 108us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:38:41.539102 (+ 90us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:41.539472 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:41.539483 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:41.539491 (+ 8us) client_negotiation.cc:769] Sending connection context
0810 11:38:41.539784 (+ 293us) client_negotiation.cc:240] Negotiation successful
0810 11:38:41.540036 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":325,"threads_started":1}
I20260810 11:38:41.542847 19064 heartbeater.cc:347] Connected to a master server at 127.3.240.254:38073
I20260810 11:38:41.543205 19064 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:41.543927 19064 heartbeater.cc:511] Master 127.3.240.254:38073 requested a full tablet report, sending...
I20260810 11:38:41.545241 18381 ts_manager.cc:194] Registered new tserver with Master: b6c837f3ff384531bf4fd974dd5f92be (127.3.240.195:42121)
I20260810 11:38:41.556784 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260810 11:38:41.563721 19066 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:38:41.561737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58268 (local address 127.3.240.254:38073)
0810 11:38:41.561925 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.561930 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.562486 (+ 556us) server_negotiation.cc:407] Connection header received
0810 11:38:41.562641 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.562645 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.562698 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.562788 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.563532 (+ 744us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58268: BlockingRecv error: recv got EOF from 127.0.0.1:58268 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 11:38:41.564584 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 18490
I20260810 11:38:41.570053 19080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.566342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39928 (local address 127.3.240.193:37411)
0810 11:38:41.567451 (+ 1109us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.567461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.567496 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:38:41.567579 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.567595 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.567803 (+ 208us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.567996 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.569226 (+ 1230us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:41.569234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:41.569248 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:41.569263 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:41.569312 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:41.569326 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:41.569558 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:41.569629 (+ 71us) server_negotiation.cc:299] Negotiation successful
0810 11:38:41.569756 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":722,"threads_started":1}
I20260810 11:38:41.571375 18668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:41.659463 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 18490
I20260810 11:38:41.684122 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 18678
I20260810 11:38:41.689172 19083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.685675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50050 (local address 127.3.240.194:40627)
0810 11:38:41.686530 (+ 855us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.686539 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.686573 (+ 34us) server_negotiation.cc:407] Connection header received
0810 11:38:41.686647 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.686653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.686977 (+ 324us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.687143 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.688330 (+ 1187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:41.688338 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:41.688351 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:41.688376 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:41.688429 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:41.688445 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:41.688570 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:41.688723 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 11:38:41.688856 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":443,"threads_started":1}
I20260810 11:38:41.690493 18852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:41.776259 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 18678
I20260810 11:38:41.801806 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 18870
I20260810 11:38:41.806521 19087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:41.803062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52618 (local address 127.3.240.195:42121)
0810 11:38:41.803675 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:41.803682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:41.803712 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:38:41.803774 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:41.803780 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:41.804007 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:41.804133 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:41.805311 (+ 1178us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:41.805318 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:41.805330 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:38:41.805343 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:41.805383 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:38:41.805397 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:41.805546 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:41.805672 (+ 126us) server_negotiation.cc:299] Negotiation successful
0810 11:38:41.805755 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":303,"threads_started":1}
I20260810 11:38:41.808256 19057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:41.900499 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 18870
I20260810 11:38:41.932160 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 18331
I20260810 11:38:41.934078 18432 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:42.051172 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 18331
2026-08-10T11:38:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2750 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:38:42 dist-test-slave-drl0 krb5kdc[19095](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:38:42 dist-test-slave-drl0 krb5kdc[19095](info): set up 2 sockets
Aug 10 11:38:42 dist-test-slave-drl0 krb5kdc[19095](info): commencing operation
krb5kdc: starting...
W20260810 11:38:43.405040 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.303s user 0.005s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:38:43 dist-test-slave-drl0 krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:38:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:38:43.739951 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41897
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:41897
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:44.123404 19114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:44.123697 19114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:44.123867 19114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:44.131541 19114 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:38:44.131654 19114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:44.131703 19114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:44.131853 19114 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:38:44.131932 19114 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:38:44.141770 19114 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:41897
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:41897
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:44.143235 19114 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:44.144575 19114 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:44.156706 19120 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:44.156741 19121 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:44.156656 19124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:44.157995 19114 server_base.cc:1034] running on GCE node
I20260810 11:38:44.158800 19114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:44.160254 19114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:44.161520 19114 hybrid_clock.cc:648] HybridClock initialized: now 1786361924161484 us; error 52 us; skew 500 ppm
I20260810 11:38:44.161906 19114 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:44 dist-test-slave-drl0 krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:44.165484 19114 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:38:44.167347 19114 webserver.cc:460] Webserver started at http://127.3.240.254:41777/ using document root <none> and password file <none>
I20260810 11:38:44.168094 19114 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:44.168274 19114 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:44.168627 19114 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:44.172163 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "c582c19fe610432695bba993e9578222"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c94fc1f8f37127a22539da4e47ff2f8b"
server_key_iv: "20e7042adaf45bb31e9081e63c5db037"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.173004 19114 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "c582c19fe610432695bba993e9578222"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c94fc1f8f37127a22539da4e47ff2f8b"
server_key_iv: "20e7042adaf45bb31e9081e63c5db037"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.179277 19114 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:38:44.183053 19133 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:44.184667 19114 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 11:38:44.184894 19114 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "c582c19fe610432695bba993e9578222"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c94fc1f8f37127a22539da4e47ff2f8b"
server_key_iv: "20e7042adaf45bb31e9081e63c5db037"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.185113 19114 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:44.203698 19114 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:44.208974 19114 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:44.209326 19114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:44.230108 19114 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:41897
I20260810 11:38:44.230146 19201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:41897 every 8 connection(s)
I20260810 11:38:44.231616 19114 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:38:44.237162 19202 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:44.238531 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 19114
I20260810 11:38:44.238814 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:38:44.239566 4035 external_mini_cluster.cc:1428] Setting key e365ebd2d95b0d880f13f0646dd505a1
I20260810 11:38:44.255409 19202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: Bootstrap starting.
Aug 10 11:38:44 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:44.258569 19202 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:44.259907 19202 log.cc:824] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:44.264762 19202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: No bootstrap required, opened a new log
I20260810 11:38:44.270063 19205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:44.242268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42712 (local address 127.3.240.254:41897)
0810 11:38:44.243448 (+ 1180us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:44.243465 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:44.243531 (+ 66us) server_negotiation.cc:407] Connection header received
0810 11:38:44.246563 (+ 3032us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:44.246601 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:44.247206 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:44.247743 (+ 537us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:44.249954 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.252221 (+ 2267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.254136 (+ 1915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.255215 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.255441 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:44.259697 (+ 4256us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:44.259710 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:44.259730 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:44.259780 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:44.264318 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:44.265228 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:44.265238 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:44.265248 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:44.265607 (+ 359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:44.266262 (+ 655us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:44.266267 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:44.266271 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:44.267168 (+ 897us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:44.267532 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:44.268088 (+ 556us) server_negotiation.cc:299] Negotiation successful
0810 11:38:44.268619 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":891,"thread_start_us":664,"threads_started":1}
I20260810 11:38:44.275480 19202 raft_consensus.cc:348] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } }
I20260810 11:38:44.275831 19202 raft_consensus.cc:374] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:44.275907 19202 raft_consensus.cc:729] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c582c19fe610432695bba993e9578222, State: Initialized, Role: FOLLOWER
I20260810 11:38:44.276602 19202 consensus_queue.cc:260] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } }
I20260810 11:38:44.276865 19202 raft_consensus.cc:388] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:38:44.276963 19202 raft_consensus.cc:482] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:38:44.277460 19202 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:44.280120 19202 raft_consensus.cc:504] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } }
I20260810 11:38:44.280634 19202 leader_election.cc:302] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c582c19fe610432695bba993e9578222; no voters:
I20260810 11:38:44.281275 19202 leader_election.cc:288] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:38:44.281423 19208 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:44.282226 19208 raft_consensus.cc:686] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [term 1 LEADER]: Becoming Leader. State: Replica: c582c19fe610432695bba993e9578222, State: Running, Role: LEADER
I20260810 11:38:44.283118 19208 consensus_queue.cc:237] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } }
I20260810 11:38:44.283720 19202 sys_catalog.cc:564] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:38:44.286324 19211 sys_catalog.cc:455] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c582c19fe610432695bba993e9578222. Latest consensus state: current_term: 1 leader_uuid: "c582c19fe610432695bba993e9578222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } } }
I20260810 11:38:44.286762 19211 sys_catalog.cc:458] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:44.287256 19210 sys_catalog.cc:455] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c582c19fe610432695bba993e9578222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c582c19fe610432695bba993e9578222" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 41897 } } }
I20260810 11:38:44.287618 19210 sys_catalog.cc:458] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222 [sys.catalog]: This master's current role is: LEADER
I20260810 11:38:44.289342 19219 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:38:44.294848 19219 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:38:44.304989 19219 catalog_manager.cc:1347] Generated new cluster ID: 0e94b43a38cd4414990e5faf74635706
I20260810 11:38:44.305150 19219 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:38:44.323395 19219 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:38:44.325011 19219 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:38:44.337173 19219 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: Generated new TSK 0
I20260810 11:38:44.338073 19219 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:38:44.431344 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:44.835439 19237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:44.835721 19237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:44.835819 19237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:44.844962 19237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:44.845142 19237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:44.845299 19237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:38:44.857062 19237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:44.858486 19237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:44.859856 19237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:44.873492 19245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:44.873570 19247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:44.873570 19244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:44.874639 19237 server_base.cc:1034] running on GCE node
I20260810 11:38:44.875644 19237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:44.876993 19237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:44.878307 19237 hybrid_clock.cc:648] HybridClock initialized: now 1786361924878223 us; error 60 us; skew 500 ppm
I20260810 11:38:44.878757 19237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:44 dist-test-slave-drl0 krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:44.882781 19237 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:38:44.884909 19237 webserver.cc:460] Webserver started at http://127.3.240.193:39773/ using document root <none> and password file <none>
I20260810 11:38:44.886130 19237 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:44.886404 19237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:44.887036 19237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:44.890385 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "0e53cdab6be84c449a0596e434413ad4"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c82292de7a36506e10bbb2618358ff95"
server_key_iv: "5da395df6bdcd27c0263e848a138b9f7"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.891292 19237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "0e53cdab6be84c449a0596e434413ad4"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c82292de7a36506e10bbb2618358ff95"
server_key_iv: "5da395df6bdcd27c0263e848a138b9f7"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.897523 19237 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:38:44.901367 19262 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:44.903122 19237 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:38:44.903314 19237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "0e53cdab6be84c449a0596e434413ad4"
format_stamp: "Formatted at 2026-08-10 11:38:44 on dist-test-slave-drl0"
server_key: "c82292de7a36506e10bbb2618358ff95"
server_key_iv: "5da395df6bdcd27c0263e848a138b9f7"
server_key_version: "encryptionkey@0"
I20260810 11:38:44.903541 19237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:44.924445 19237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:44.930574 19237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:44.930985 19237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:44.932432 19237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:44.934728 19237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:44.934834 19237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:44.935011 19237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:44.935077 19237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:44.981244 19237 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:41345
I20260810 11:38:44.981297 19417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:41345 every 8 connection(s)
I20260810 11:38:44.982674 19237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:38:44.984053 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 19237
I20260810 11:38:44.984313 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:38:44.985023 4035 external_mini_cluster.cc:1428] Setting key e208b8f4501c7a443a91984ba972d5bf
Aug 10 11:38:44 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361924, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:45.008472 19420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:44.986511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50419 (local address 127.3.240.254:41897)
0810 11:38:44.989284 (+ 2773us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:44.989290 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:44.989312 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:38:44.990948 (+ 1636us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:44.990955 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:44.991090 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:44.991232 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:44.993011 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.993754 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.995398 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.995789 (+ 391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.995936 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:45.000689 (+ 4753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:45.000698 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:45.000705 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:45.000747 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:45.004395 (+ 3648us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:45.005499 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:45.005508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:45.005515 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:45.005706 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:45.006227 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:45.006235 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:45.006240 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:45.006675 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:45.006846 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:45.007650 (+ 804us) server_negotiation.cc:299] Negotiation successful
0810 11:38:45.008001 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2649,"thread_start_us":2540,"threads_started":1}
I20260810 11:38:45.009167 19421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:44.986914 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41897 (local address 127.3.240.193:50419)
0810 11:38:44.988340 (+ 1426us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:44.988413 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:44.990628 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:44.991493 (+ 865us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:44.991502 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:44.992032 (+ 530us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:44.992821 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:44.992844 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.993942 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.993946 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:44.995229 (+ 1283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:44.995242 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:44.996055 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:44.996062 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:44.996289 (+ 227us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:44.997013 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:44.997041 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:45.000335 (+ 3294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:45.004743 (+ 4408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:45.004755 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:45.004773 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:45.005317 (+ 544us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:45.005867 (+ 550us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:45.005871 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:45.005874 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:45.006086 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:45.006813 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:45.006819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:45.007231 (+ 412us) client_negotiation.cc:769] Sending connection context
0810 11:38:45.007479 (+ 248us) client_negotiation.cc:240] Negotiation successful
0810 11:38:45.007976 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1043,"thread_start_us":731,"threads_started":1}
I20260810 11:38:45.010947 19418 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41897
I20260810 11:38:45.011318 19418 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:45.012275 19418 heartbeater.cc:511] Master 127.3.240.254:41897 requested a full tablet report, sending...
I20260810 11:38:45.015023 19155 ts_manager.cc:194] Registered new tserver with Master: 0e53cdab6be84c449a0596e434413ad4 (127.3.240.193:41345)
I20260810 11:38:45.016553 19155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:50419
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:38:45.071656 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:38:45.441566 19425 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:45.441839 19425 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:45.441980 19425 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:45.450096 19425 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:45.450212 19425 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:45.450402 19425 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:38:45.460536 19425 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:45.461885 19425 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:45.463265 19425 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:45.477559 19432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:45.477602 19434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:45.477645 19437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:45.479082 19425 server_base.cc:1034] running on GCE node
I20260810 11:38:45.479660 19425 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:45.480588 19425 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:45.481838 19425 hybrid_clock.cc:648] HybridClock initialized: now 1786361925481774 us; error 40 us; skew 500 ppm
I20260810 11:38:45.482193 19425 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:45 dist-test-slave-drl0 krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:45.485877 19425 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:38:45.487669 19425 webserver.cc:460] Webserver started at http://127.3.240.194:43609/ using document root <none> and password file <none>
I20260810 11:38:45.488463 19425 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:45.488619 19425 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:45.489065 19425 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:45.492322 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "c59146939fb2494c8406dbbd646c1ae5"
format_stamp: "Formatted at 2026-08-10 11:38:45 on dist-test-slave-drl0"
server_key: "ed86890c4b665b030ed8c4f57c9d7bd9"
server_key_iv: "4805d416feb33d77c58475250741ae52"
server_key_version: "encryptionkey@0"
I20260810 11:38:45.493089 19425 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "c59146939fb2494c8406dbbd646c1ae5"
format_stamp: "Formatted at 2026-08-10 11:38:45 on dist-test-slave-drl0"
server_key: "ed86890c4b665b030ed8c4f57c9d7bd9"
server_key_iv: "4805d416feb33d77c58475250741ae52"
server_key_version: "encryptionkey@0"
I20260810 11:38:45.498459 19425 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 11:38:45.503379 19445 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:45.505004 19425 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260810 11:38:45.505213 19425 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "c59146939fb2494c8406dbbd646c1ae5"
format_stamp: "Formatted at 2026-08-10 11:38:45 on dist-test-slave-drl0"
server_key: "ed86890c4b665b030ed8c4f57c9d7bd9"
server_key_iv: "4805d416feb33d77c58475250741ae52"
server_key_version: "encryptionkey@0"
I20260810 11:38:45.505450 19425 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:45.530887 19425 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:45.535821 19425 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:45.536175 19425 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:45.537397 19425 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:45.539417 19425 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:45.539541 19425 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:45.539724 19425 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:45.539804 19425 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:45.580421 19425 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:39041
I20260810 11:38:45.580449 19613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:39041 every 8 connection(s)
I20260810 11:38:45.581753 19425 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:38:45.590525 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 19425
I20260810 11:38:45.590751 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:38:45.591344 4035 external_mini_cluster.cc:1428] Setting key c7aca326614c712924f2eedf56b751f3
Aug 10 11:38:45 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361925, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:45.607558 19617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:45.584707 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:51987 (local address 127.3.240.254:41897)
0810 11:38:45.585175 (+ 468us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:45.585179 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:45.586694 (+ 1515us) server_negotiation.cc:407] Connection header received
0810 11:38:45.588613 (+ 1919us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:45.588620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:45.588712 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:45.588838 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:45.590409 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:45.591338 (+ 929us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:45.593392 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:45.593920 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:45.594031 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:45.598453 (+ 4422us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:45.598459 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:45.598462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:45.598493 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:45.602832 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:45.603855 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:45.603863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:45.603869 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:45.604032 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:45.604570 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:45.604578 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:45.604583 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:45.604994 (+ 411us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:45.605139 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:45.606790 (+ 1651us) server_negotiation.cc:299] Negotiation successful
0810 11:38:45.607069 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":252,"threads_started":1}
I20260810 11:38:45.609532 19618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:45.585565 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41897 (local address 127.3.240.194:51987)
0810 11:38:45.586515 (+ 950us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:45.586564 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:45.588346 (+ 1782us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:45.589061 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:45.589069 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:45.589647 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:45.590233 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:45.590252 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:45.591487 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:45.591493 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:45.593064 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:45.593079 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:45.594180 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:45.594188 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:45.594505 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:45.595334 (+ 829us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:45.595369 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:45.598198 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:45.603025 (+ 4827us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:45.603038 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:45.603057 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:45.603696 (+ 639us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:45.604168 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:45.604175 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:45.604181 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:45.604430 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:45.605216 (+ 786us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:45.605227 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:45.605693 (+ 466us) client_negotiation.cc:769] Sending connection context
0810 11:38:45.606009 (+ 316us) client_negotiation.cc:240] Negotiation successful
0810 11:38:45.607951 (+ 1942us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":360,"threads_started":1}
I20260810 11:38:45.611295 19615 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41897
I20260810 11:38:45.611567 19615 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:45.612334 19615 heartbeater.cc:511] Master 127.3.240.254:41897 requested a full tablet report, sending...
I20260810 11:38:45.614090 19155 ts_manager.cc:194] Registered new tserver with Master: c59146939fb2494c8406dbbd646c1ae5 (127.3.240.194:39041)
I20260810 11:38:45.615059 19155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:51987
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:38:45.670531 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:38:46.020480 19418 heartbeater.cc:503] Master 127.3.240.254:41897 was elected leader, sending a full tablet report...
W20260810 11:38:46.043422 19624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:38:46.043752 19624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:38:46.043891 19624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:38:46.053292 19624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:38:46.053413 19624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:38:46.053596 19624 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:38:46.063921 19624 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:41897
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:38:46.065338 19624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:38:46.066727 19624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:38:46.082245 19633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:46.082442 19635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:38:46.082995 19632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:38:46.083542 19624 server_base.cc:1034] running on GCE node
I20260810 11:38:46.084326 19624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:38:46.085507 19624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:38:46.086910 19624 hybrid_clock.cc:648] HybridClock initialized: now 1786361926086851 us; error 72 us; skew 500 ppm
I20260810 11:38:46.087323 19624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:38:46 dist-test-slave-drl0 krb5kdc[19095](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361926, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:38:46.090835 19624 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:38:46.092448 19624 webserver.cc:460] Webserver started at http://127.3.240.195:33193/ using document root <none> and password file <none>
I20260810 11:38:46.093189 19624 fs_manager.cc:359] Metadata directory not provided
I20260810 11:38:46.093299 19624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:38:46.093664 19624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:38:46.096724 19624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "448fe7d9f82744e7a706d731627e3e8a"
format_stamp: "Formatted at 2026-08-10 11:38:46 on dist-test-slave-drl0"
server_key: "bdfa5175d6118bae2ef23185e20d8812"
server_key_iv: "1b41bbf9cabf2c4db2bcd3c464cf18cc"
server_key_version: "encryptionkey@0"
I20260810 11:38:46.097990 19624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "448fe7d9f82744e7a706d731627e3e8a"
format_stamp: "Formatted at 2026-08-10 11:38:46 on dist-test-slave-drl0"
server_key: "bdfa5175d6118bae2ef23185e20d8812"
server_key_iv: "1b41bbf9cabf2c4db2bcd3c464cf18cc"
server_key_version: "encryptionkey@0"
I20260810 11:38:46.103825 19624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:38:46.107733 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:46.109378 19624 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 11:38:46.109529 19624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "448fe7d9f82744e7a706d731627e3e8a"
format_stamp: "Formatted at 2026-08-10 11:38:46 on dist-test-slave-drl0"
server_key: "bdfa5175d6118bae2ef23185e20d8812"
server_key_iv: "1b41bbf9cabf2c4db2bcd3c464cf18cc"
server_key_version: "encryptionkey@0"
I20260810 11:38:46.109761 19624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:38:46.128055 19624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:38:46.133440 19624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:38:46.133800 19624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:38:46.134897 19624 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:38:46.136420 19624 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:38:46.136497 19624 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:46.136632 19624 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:38:46.136686 19624 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:38:46.178189 19624 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:36893
I20260810 11:38:46.178267 19815 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:36893 every 8 connection(s)
I20260810 11:38:46.180063 19624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:38:46.185730 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 19624
I20260810 11:38:46.185957 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:38:46.186569 4035 external_mini_cluster.cc:1428] Setting key 97d07b5ffc3ba18404d81bafc827a238
Aug 10 11:38:46 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361926, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:38:46.204351 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.183853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:41829 (local address 127.3.240.254:41897)
0810 11:38:46.184377 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.184383 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.185562 (+ 1179us) server_negotiation.cc:407] Connection header received
0810 11:38:46.188110 (+ 2548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.188115 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.188227 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.188353 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:46.190076 (+ 1723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.191029 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.192802 (+ 1773us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.193207 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.193353 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.196964 (+ 3611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:46.196969 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:46.196972 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:46.197005 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:46.199974 (+ 2969us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.200906 (+ 932us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.200913 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.200918 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.201096 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.201976 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.201980 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.201982 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.202639 (+ 657us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:46.202881 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.203709 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.204032 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":384,"thread_start_us":283,"threads_started":1}
I20260810 11:38:46.205065 19821 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.184472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:41897 (local address 127.3.240.195:41829)
0810 11:38:46.185360 (+ 888us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.185424 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.187814 (+ 2390us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.188601 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.188610 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.189303 (+ 693us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:38:46.189884 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.189901 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.191225 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.191233 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.192622 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.192636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.193452 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.193460 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.193708 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.194437 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:38:46.194460 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:38:46.196742 (+ 2282us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:38:46.200146 (+ 3404us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:46.200156 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:46.200172 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:46.200750 (+ 578us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:46.201559 (+ 809us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:38:46.201563 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:38:46.201566 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:38:46.201828 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:38:46.202860 (+ 1032us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:38:46.202867 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:38:46.203277 (+ 410us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.203548 (+ 271us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.203948 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":315,"threads_started":1}
I20260810 11:38:46.206540 19816 heartbeater.cc:347] Connected to a master server at 127.3.240.254:41897
I20260810 11:38:46.207019 19816 heartbeater.cc:464] Registering TS with master...
I20260810 11:38:46.207808 19816 heartbeater.cc:511] Master 127.3.240.254:41897 requested a full tablet report, sending...
I20260810 11:38:46.209353 19155 ts_manager.cc:194] Registered new tserver with Master: 448fe7d9f82744e7a706d731627e3e8a (127.3.240.195:36893)
I20260810 11:38:46.210346 19155 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:41829
I20260810 11:38:46.216686 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:38:46.238219 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.221994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42724 (local address 127.3.240.254:41897)
0810 11:38:46.222327 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.222338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.223079 (+ 741us) server_negotiation.cc:407] Connection header received
0810 11:38:46.223261 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.223265 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.223335 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.223430 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:46.224934 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.225689 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.227038 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.227565 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.227670 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.230822 (+ 3152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:46.230829 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:46.230834 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:46.230875 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:46.234591 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.235439 (+ 848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.235445 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.235450 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.235581 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.236001 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.236008 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.236012 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.236363 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:46.236873 (+ 510us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.237431 (+ 558us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.237773 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 11:38:46.243996 19155 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:38:46.250844 19155 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:38:46.289597 19835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.265305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41345 (local address 127.0.0.1:38002)
0810 11:38:46.278623 (+ 13318us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.278663 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.278894 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.280727 (+ 1833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.280737 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.280769 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:46.281201 (+ 432us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.281212 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.283336 (+ 2124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.283342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.286584 (+ 3242us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.286605 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.288417 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.288424 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.288616 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.288682 (+ 66us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.288828 (+ 146us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.289030 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13062,"mutex_wait_us":94,"thread_start_us":13400,"threads_started":1}
I20260810 11:38:46.290392 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.266152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35542 (local address 127.3.240.195:36893)
0810 11:38:46.279683 (+ 13531us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.279695 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.280595 (+ 900us) server_negotiation.cc:407] Connection header received
0810 11:38:46.280814 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.280822 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.281046 (+ 224us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.281216 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:46.282192 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.283772 (+ 1580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.287180 (+ 3408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.288631 (+ 1451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.288747 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.288957 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.289858 (+ 901us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.290046 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13337,"thread_start_us":13160,"threads_started":1}
I20260810 11:38:46.290463 19834 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.265405 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:39041 (local address 127.0.0.1:40340)
0810 11:38:46.279023 (+ 13618us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.279047 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.279837 (+ 790us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.281310 (+ 1473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.281314 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.281343 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:46.281753 (+ 410us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.281763 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.283673 (+ 1910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.283679 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.286711 (+ 3032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.286726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.289106 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.289113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.289305 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.289334 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.289577 (+ 243us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.289729 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13461,"mutex_wait_us":115,"thread_start_us":12813,"threads_started":1}
I20260810 11:38:46.290572 19838 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.265863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36893 (local address 127.0.0.1:35542)
0810 11:38:46.280460 (+ 14597us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.280478 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.280602 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.281473 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.281477 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.281506 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:46.281994 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.282006 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.284350 (+ 2344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.284358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.287010 (+ 2652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.287024 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.288842 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.288847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.289002 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.289038 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.289129 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.289283 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14488,"mutex_wait_us":5,"thread_start_us":15083,"threads_started":1}
I20260810 11:38:46.290673 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.265951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38002 (local address 127.3.240.193:41345)
0810 11:38:46.280231 (+ 14280us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.280236 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.280258 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:38:46.280331 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.280336 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.280490 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.280707 (+ 217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:46.281395 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.283133 (+ 1738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.286790 (+ 3657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.288182 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.288363 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.288616 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.290139 (+ 1523us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.290324 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14139,"thread_start_us":13970,"threads_started":1}
I20260810 11:38:46.290992 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.265469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40340 (local address 127.3.240.194:39041)
0810 11:38:46.277814 (+ 12345us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.277828 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.279148 (+ 1320us) server_negotiation.cc:407] Connection header received
0810 11:38:46.280219 (+ 1071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.280231 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.280516 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.280688 (+ 172us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:46.281943 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.283540 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.287013 (+ 3473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.288580 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.288730 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.288921 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.289618 (+ 697us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.290102 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12099,"thread_start_us":12377,"threads_started":1}
I20260810 11:38:46.294171 19718 tablet_service.cc:1467] Processing CreateTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 (DEFAULT_TABLE table=test-table [id=ee939408c9bc4718a6523fd3c706cd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:46.294175 19519 tablet_service.cc:1467] Processing CreateTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 (DEFAULT_TABLE table=test-table [id=ee939408c9bc4718a6523fd3c706cd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:46.295594 19519 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7185ebcd68f44ba5aeac88124f06b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:46.295660 19718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7185ebcd68f44ba5aeac88124f06b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:46.297511 19327 tablet_service.cc:1467] Processing CreateTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 (DEFAULT_TABLE table=test-table [id=ee939408c9bc4718a6523fd3c706cd94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:38:46.298960 19327 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7185ebcd68f44ba5aeac88124f06b002. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:38:46.307127 19842 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Bootstrap starting.
I20260810 11:38:46.311503 19842 tablet_bootstrap.cc:654] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:46.311690 19843 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Bootstrap starting.
I20260810 11:38:46.312997 19842 log.cc:824] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:46.314146 19844 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Bootstrap starting.
I20260810 11:38:46.315623 19843 tablet_bootstrap.cc:654] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:46.316596 19843 log.cc:824] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:46.319036 19842 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: No bootstrap required, opened a new log
I20260810 11:38:46.319341 19842 ts_tablet_manager.cc:1397] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.003s
I20260810 11:38:46.319694 19844 tablet_bootstrap.cc:654] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Neither blocks nor log segments found. Creating new log.
I20260810 11:38:46.319916 19843 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: No bootstrap required, opened a new log
I20260810 11:38:46.320168 19843 ts_tablet_manager.cc:1397] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.001s
I20260810 11:38:46.320978 19844 log.cc:824] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Log is configured to *not* fsync() on all Append() calls
I20260810 11:38:46.324108 19844 tablet_bootstrap.cc:492] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: No bootstrap required, opened a new log
I20260810 11:38:46.324368 19844 ts_tablet_manager.cc:1397] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 11:38:46.327663 19843 raft_consensus.cc:348] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.328043 19843 raft_consensus.cc:374] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:46.328148 19843 raft_consensus.cc:729] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 448fe7d9f82744e7a706d731627e3e8a, State: Initialized, Role: FOLLOWER
I20260810 11:38:46.328785 19843 consensus_queue.cc:260] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.330403 19816 heartbeater.cc:503] Master 127.3.240.254:41897 was elected leader, sending a full tablet report...
I20260810 11:38:46.330174 19842 raft_consensus.cc:348] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.330595 19842 raft_consensus.cc:374] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:46.330683 19842 raft_consensus.cc:729] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c59146939fb2494c8406dbbd646c1ae5, State: Initialized, Role: FOLLOWER
I20260810 11:38:46.331128 19843 ts_tablet_manager.cc:1428] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20260810 11:38:46.331310 19842 consensus_queue.cc:260] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.332564 19615 heartbeater.cc:503] Master 127.3.240.254:41897 was elected leader, sending a full tablet report...
I20260810 11:38:46.333153 19842 ts_tablet_manager.cc:1428] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
W20260810 11:38:46.334488 19616 tablet.cc:2369] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:38:46.336504 19844 raft_consensus.cc:348] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.336943 19844 raft_consensus.cc:374] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:38:46.337070 19844 raft_consensus.cc:729] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e53cdab6be84c449a0596e434413ad4, State: Initialized, Role: FOLLOWER
I20260810 11:38:46.337733 19844 consensus_queue.cc:260] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.339231 19844 ts_tablet_manager.cc:1428] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
W20260810 11:38:46.432600 19818 tablet.cc:2369] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:38:46.485764 19419 tablet.cc:2369] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:38:46.513775 19851 raft_consensus.cc:482] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:38:46.514143 19851 raft_consensus.cc:504] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.516215 19851 leader_election.cc:288] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0e53cdab6be84c449a0596e434413ad4 (127.3.240.193:41345), 448fe7d9f82744e7a706d731627e3e8a (127.3.240.195:36893)
I20260810 11:38:46.524096 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.516956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:41523 (local address 127.3.240.195:36893)
0810 11:38:46.517213 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.517221 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.517869 (+ 648us) server_negotiation.cc:407] Connection header received
0810 11:38:46.518216 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.518221 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.518292 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.518357 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:46.519473 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.520263 (+ 790us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.522301 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.523135 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.523336 (+ 201us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.523414 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.523711 (+ 297us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.523836 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 11:38:46.524250 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.516542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36893 (local address 127.3.240.194:41523)
0810 11:38:46.517772 (+ 1230us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.517790 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.518045 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.518587 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.518591 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.518616 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:46.519320 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.519330 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.520430 (+ 1100us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.520438 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.522128 (+ 1690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.522142 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.523358 (+ 1216us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.523365 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.523542 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.523566 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.523675 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.523819 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1122,"mutex_wait_us":57,"thread_start_us":548,"threads_started":1}
I20260810 11:38:46.524721 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.516510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:41345 (local address 127.3.240.194:54089)
0810 11:38:46.517484 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0810 11:38:46.517507 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:38:46.517739 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:38:46.518169 (+ 430us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:38:46.518174 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:38:46.518203 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:38:46.518605 (+ 402us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.518618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.520315 (+ 1697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.520323 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.522335 (+ 2012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:38:46.522363 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.523825 (+ 1462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.523832 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:38:46.524019 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.524044 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:38:46.524160 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 11:38:46.524326 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":810,"mutex_wait_us":53,"thread_start_us":981,"threads_started":1}
I20260810 11:38:46.524994 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.516615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54089 (local address 127.3.240.193:41345)
0810 11:38:46.516878 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.516887 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.517610 (+ 723us) server_negotiation.cc:407] Connection header received
0810 11:38:46.517930 (+ 320us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.517935 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.518023 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.518146 (+ 123us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:38:46.519231 (+ 1085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.520168 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.522698 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.523583 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.523773 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.523864 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.524283 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.524414 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:38:46.525358 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7185ebcd68f44ba5aeac88124f06b002" candidate_uuid: "c59146939fb2494c8406dbbd646c1ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448fe7d9f82744e7a706d731627e3e8a" is_pre_election: true
I20260810 11:38:46.526361 19746 raft_consensus.cc:2454] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59146939fb2494c8406dbbd646c1ae5 in term 0.
I20260810 11:38:46.526134 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7185ebcd68f44ba5aeac88124f06b002" candidate_uuid: "c59146939fb2494c8406dbbd646c1ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53cdab6be84c449a0596e434413ad4" is_pre_election: true
I20260810 11:38:46.526975 19354 raft_consensus.cc:2454] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c59146939fb2494c8406dbbd646c1ae5 in term 0.
I20260810 11:38:46.527349 19447 leader_election.cc:302] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448fe7d9f82744e7a706d731627e3e8a, c59146939fb2494c8406dbbd646c1ae5; no voters:
I20260810 11:38:46.527815 19851 raft_consensus.cc:2781] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:38:46.527984 19851 raft_consensus.cc:482] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:38:46.528100 19851 raft_consensus.cc:3037] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:46.530432 19851 raft_consensus.cc:504] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.531101 19851 leader_election.cc:288] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [CANDIDATE]: Term 1 election: Requested vote from peers 0e53cdab6be84c449a0596e434413ad4 (127.3.240.193:41345), 448fe7d9f82744e7a706d731627e3e8a (127.3.240.195:36893)
I20260810 11:38:46.531594 19746 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7185ebcd68f44ba5aeac88124f06b002" candidate_uuid: "c59146939fb2494c8406dbbd646c1ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "448fe7d9f82744e7a706d731627e3e8a"
I20260810 11:38:46.531657 19354 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7185ebcd68f44ba5aeac88124f06b002" candidate_uuid: "c59146939fb2494c8406dbbd646c1ae5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0e53cdab6be84c449a0596e434413ad4"
I20260810 11:38:46.531772 19746 raft_consensus.cc:3037] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:46.531939 19354 raft_consensus.cc:3037] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:38:46.534065 19746 raft_consensus.cc:2454] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59146939fb2494c8406dbbd646c1ae5 in term 1.
I20260810 11:38:46.534124 19354 raft_consensus.cc:2454] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c59146939fb2494c8406dbbd646c1ae5 in term 1.
I20260810 11:38:46.534479 19447 leader_election.cc:302] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 448fe7d9f82744e7a706d731627e3e8a, c59146939fb2494c8406dbbd646c1ae5; no voters:
I20260810 11:38:46.535174 19851 raft_consensus.cc:2781] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:38:46.536098 19851 raft_consensus.cc:686] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 1 LEADER]: Becoming Leader. State: Replica: c59146939fb2494c8406dbbd646c1ae5, State: Running, Role: LEADER
I20260810 11:38:46.536765 19851 consensus_queue.cc:237] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } }
I20260810 11:38:46.542891 19155 catalog_manager.cc:5495] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 reported cstate change: term changed from 0 to 1, leader changed from <none> to c59146939fb2494c8406dbbd646c1ae5 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "c59146939fb2494c8406dbbd646c1ae5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c59146939fb2494c8406dbbd646c1ae5" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 39041 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260810 11:38:46.593688 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.585735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40350 (local address 127.3.240.194:39041)
0810 11:38:46.585951 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.585958 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.586076 (+ 118us) server_negotiation.cc:407] Connection header received
0810 11:38:46.586353 (+ 277us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.586362 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.586462 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.586627 (+ 165us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:38:46.587127 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.588504 (+ 1377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.590037 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.590513 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.590651 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.591215 (+ 564us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:38:46.591737 (+ 522us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:38:46.591995 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.593025 (+ 1030us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.593115 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:38:46.604584 19746 raft_consensus.cc:1261] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 1 FOLLOWER]: Refusing update from remote peer c59146939fb2494c8406dbbd646c1ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:46.604875 19354 raft_consensus.cc:1261] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 1 FOLLOWER]: Refusing update from remote peer c59146939fb2494c8406dbbd646c1ae5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:38:46.605719 19851 consensus_queue.cc:1035] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0e53cdab6be84c449a0596e434413ad4" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 41345 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:46.606086 19858 consensus_queue.cc:1035] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "448fe7d9f82744e7a706d731627e3e8a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:38:46.630431 19861 mvcc.cc:205] Tried to move back new op lower bound from 7316938451360493568 to 7316938451099516928. Current Snapshot: MvccSnapshot[applied={T|T < 7316938451360493568 or (T in {7316938451360493568})}]
I20260810 11:38:46.639012 19154 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:46.639386 19154 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42724:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:38:46.646827 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:46 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35542
I20260810 11:38:46.647763 19154 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c582c19fe610432695bba993e9578222: Sending DeleteTablet for 3 replicas of tablet 7185ebcd68f44ba5aeac88124f06b002
I20260810 11:38:46.647837 19872 tablet_replica.cc:330] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: stopping tablet replica
I20260810 11:38:46.648264 19872 raft_consensus.cc:2229] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:46.648653 19519 tablet_service.cc:1514] Processing DeleteTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:46 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40340
I20260810 11:38:46.648788 19872 raft_consensus.cc:2258] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:46.649731 19873 tablet_replica.cc:330] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: stopping tablet replica
I20260810 11:38:46.650161 19873 raft_consensus.cc:2229] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:38:46.650738 19327 tablet_service.cc:1514] Processing DeleteTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:46 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:38002
I20260810 11:38:46.650900 19873 raft_consensus.cc:2258] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:46.652323 19874 tablet_replica.cc:330] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: stopping tablet replica
I20260810 11:38:46.652904 19874 raft_consensus.cc:2229] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:38:46.653637 19874 raft_consensus.cc:2258] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:38:46.653893 19873 ts_tablet_manager.cc:1905] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:46.654861 19718 tablet_service.cc:1514] Processing DeleteTablet for tablet 7185ebcd68f44ba5aeac88124f06b002 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:38:46 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35542
I20260810 11:38:46.655619 19874 ts_tablet_manager.cc:1905] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:46.657464 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 19237
I20260810 11:38:46.662846 19873 ts_tablet_manager.cc:1918] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:46.663303 19873 log.cc:1196] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/7185ebcd68f44ba5aeac88124f06b002
I20260810 11:38:46.663357 19872 ts_tablet_manager.cc:1905] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:38:46.663594 19874 ts_tablet_manager.cc:1918] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:46.663789 19874 log.cc:1196] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/7185ebcd68f44ba5aeac88124f06b002
I20260810 11:38:46.664371 19873 ts_tablet_manager.cc:1939] T 7185ebcd68f44ba5aeac88124f06b002 P c59146939fb2494c8406dbbd646c1ae5: Deleting consensus metadata
I20260810 11:38:46.664660 19874 ts_tablet_manager.cc:1939] T 7185ebcd68f44ba5aeac88124f06b002 P 0e53cdab6be84c449a0596e434413ad4: Deleting consensus metadata
I20260810 11:38:46.667378 19136 catalog_manager.cc:4841] TS c59146939fb2494c8406dbbd646c1ae5 (127.3.240.194:39041): tablet 7185ebcd68f44ba5aeac88124f06b002 (table test-table [id=ee939408c9bc4718a6523fd3c706cd94]) successfully deleted
I20260810 11:38:46.667928 19140 catalog_manager.cc:4841] TS 0e53cdab6be84c449a0596e434413ad4 (127.3.240.193:41345): tablet 7185ebcd68f44ba5aeac88124f06b002 (table test-table [id=ee939408c9bc4718a6523fd3c706cd94]) successfully deleted
I20260810 11:38:46.684792 19872 ts_tablet_manager.cc:1918] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:38:46.684970 19872 log.cc:1196] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/7185ebcd68f44ba5aeac88124f06b002
I20260810 11:38:46.685442 19872 ts_tablet_manager.cc:1939] T 7185ebcd68f44ba5aeac88124f06b002 P 448fe7d9f82744e7a706d731627e3e8a: Deleting consensus metadata
W20260810 11:38:46.687762 19137 catalog_manager.cc:4808] TS 448fe7d9f82744e7a706d731627e3e8a (127.3.240.195:36893): delete failed for tablet 7185ebcd68f44ba5aeac88124f06b002 because the tablet was not found. No further retry: Not found: Tablet not found: 7185ebcd68f44ba5aeac88124f06b002
Aug 10 11:38:46 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:38:46.698998 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.658812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38006 (local address 127.3.240.193:41345)
0810 11:38:46.658995 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.659001 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.677701 (+ 18700us) server_negotiation.cc:407] Connection header received
0810 11:38:46.678272 (+ 571us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.678281 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.678392 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.678481 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:46.683112 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.684052 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.686261 (+ 2209us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.686993 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.687102 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.691781 (+ 4679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:46.691797 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:46.691827 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:46.691872 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:46.695743 (+ 3871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.696472 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.696483 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.696489 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.696663 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.697117 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.697122 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.697126 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.697731 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:46.698082 (+ 351us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.698336 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.698566 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260810 11:38:46.699680 19411 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:46.793344 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 19237
I20260810 11:38:46.827423 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 19425
Aug 10 11:38:46 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:38:46.848168 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:46.828257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40358 (local address 127.3.240.194:39041)
0810 11:38:46.828467 (+ 210us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:46.828474 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:46.828526 (+ 52us) server_negotiation.cc:407] Connection header received
0810 11:38:46.828710 (+ 184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:46.828717 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:46.828814 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:46.828953 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:46.830688 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.832013 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.833518 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:46.834224 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:46.834382 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:46.839289 (+ 4907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:46.839305 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:46.839326 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:46.839386 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:46.844436 (+ 5050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.845407 (+ 971us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.845417 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.845420 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.845604 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:46.846031 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:46.846036 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:46.846039 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:46.846513 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:46.846866 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:46.847520 (+ 654us) server_negotiation.cc:299] Negotiation successful
0810 11:38:46.847764 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 11:38:46.848991 19609 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:46.982159 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 19425
I20260810 11:38:47.012645 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 19624
Aug 10 11:38:47 dist-test-slave-drl0 krb5kdc[19095](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361923, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:38:47.029176 19841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:38:47.013494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35558 (local address 127.3.240.195:36893)
0810 11:38:47.013705 (+ 211us) server_negotiation.cc:206] Beginning negotiation
0810 11:38:47.013710 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:38:47.013733 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:38:47.013926 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:38:47.013932 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:38:47.014013 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:38:47.014126 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:38:47.015545 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:47.016282 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:47.017721 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:38:47.018141 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:38:47.018282 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:38:47.023179 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:38:47.023193 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:38:47.023210 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:38:47.023249 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:38:47.026089 (+ 2840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:47.026901 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:47.026910 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:47.026916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:47.027120 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:38:47.027538 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:38:47.027543 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:38:47.027547 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:38:47.027945 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:38:47.028261 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0810 11:38:47.028581 (+ 320us) server_negotiation.cc:299] Negotiation successful
0810 11:38:47.028805 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:38:47.029893 19808 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:47.126605 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 19624
I20260810 11:38:47.152823 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 19114
I20260810 11:38:47.154470 19197 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:38:47.274036 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 19114
2026-08-10T11:38:47Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5230 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:38:47 dist-test-slave-drl0 krb5kdc[19891](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:38:47 dist-test-slave-drl0 krb5kdc[19891](info): set up 2 sockets
Aug 10 11:38:47 dist-test-slave-drl0 krb5kdc[19891](info): commencing operation
krb5kdc: starting...
W20260810 11:38:48.678853 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.005s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:38:48 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361928, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:38:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:38:48Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
I20260810 11:38:49.191576 4035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskagqWOB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/postgres -l logfile start
2026-08-10 11:38:52.135 UTC [19936] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 11:38:52.135 UTC [19936] LOG: listening on IPv4 address "127.3.240.212", port 37785
2026-08-10 11:38:52.139 UTC [19936] LOG: listening on Unix socket "/tmp/.s.PGSQL.37785"
2026-08-10 11:38:52.166 UTC [19941] LOG: database system was shut down at 2026-08-10 11:38:50 UTC
2026-08-10 11:38:52.170 UTC [19936] LOG: database system is ready to accept connections
I20260810 11:38:53.395707 4035 mini_postgres.cc:96] Postgres bound to 37785
2026-08-10 11:38:53.427 UTC [19954] FATAL: database "slave" does not exist
127.3.240.212:37785 - accepting connections
I20260810 11:38:53.428292 4035 mini_ranger.cc:161] Starting Ranger...
I20260810 11:38:53.463958 4035 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 11:38:54.061362 4035 mini_ranger.cc:87] Created ranger Postgres database
I20260810 11:38:54.061559 4035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin
2026-08-10 11:38:54,421 [I] DB FLAVOR :POSTGRES
2026-08-10 11:38:54,422 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 11:38:54,422 [I] Checking connection..
2026-08-10 11:38:54,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 11:38:54,725 [I] Checking connection passed.
2026-08-10 11:38:54,725 [I] --------- Verifying version history table ---------
2026-08-10 11:38:54,725 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:38:54,994 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 11:38:54,995 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:38:55,286 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 11:38:55,287 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 11:38:55,287 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 11:38:55.575 UTC [20044] WARNING: there is no transaction in progress
2026-08-10 11:38:55,584 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 11:38:55,584 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:38:55,892 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 11:38:55,893 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 11:38:55,893 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 11:38:56,189 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 11:38:56,487 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-drl0.c.gcp-upstream.internal','N') ;"
2026-08-10 11:38:56,782 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 11:38:57,069 [I] Table x_portal_user does not exist in database ranger
2026-08-10 11:38:57,069 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 11:38:57,368 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 11:38:57,369 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 11:38:57,657 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 11:38:57,927 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 11:38:57,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 11:38:59.058 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.078 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.100 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.119 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.138 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.160 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.222 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.230 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.246 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.259 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.270 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.286 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.297 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.305 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.313 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:38:59.322 UTC [20206] WARNING: there is no transaction in progress
2026-08-10 11:39:00,083 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 11:39:00,083 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 11:39:00,365 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 11:39:00,365 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 11:39:00,644 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 11:39:00,645 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 11:39:00,948 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 11:39:00,948 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 11:39:01,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 11:39:01,546 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-drl0.c.gcp-upstream.internal';"
2026-08-10 11:39:01,813 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 11:39:01,813 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 11:39:02,088 [I] DB_PATCHES have already been applied
I20260810 11:39:02.095045 4035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskagqWOB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 11:39:02.095171 4035 mini_ranger.cc:193] Using host: 127.3.240.212
I20260810 11:39:02.121235 4035 mini_ranger.cc:227] Ranger admin URL: http://127.3.240.212:41631
Aug 10, 2026 11:39:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:39:02 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:39:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:39:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:39:02 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:39:03 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:39:03 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 11:39:03.516968 4035 mini_ranger.cc:160] Time spent starting Ranger: real 10.089s user 0.013s sys 0.063s
Aug 10, 2026 11:39:26 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:39:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:39:26 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:39:26 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:39:27 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 11:39:27.866369 4035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 11:39:28 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361968, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 11:39:28.115772 4035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 11:39:28.147644 4035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 11:39:29.196118 4035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 11:39:29.196306 4035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms
I20260810 11:39:29.196452 4035 mini_ranger_kms.cc:227] Using postgres at 127.3.240.212:37785
2026-08-10 11:39:29,953 [I] DB FLAVOR :POSTGRES
2026-08-10 11:39:29,953 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 11:39:29,953 [I] Checking connection
2026-08-10 11:39:29,953 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 11:39:30,252 [I] connection success
2026-08-10 11:39:30,252 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 11:39:30,253 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 11:39:30,253 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 11:39:30,527 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 11:39:30,527 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 11:39:30,528 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 11:39:30,528 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:37785/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 11:39:30,833 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 11:39:31.383939 4035 mini_ranger_kms.cc:310] Created kms service
I20260810 11:39:31.566699 4035 mini_ranger_kms.cc:326] Created kudu user
I20260810 11:39:31.621881 4035 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 11:39:31 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361968, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:32.065665 4035 mini_ranger_kms.cc:384] Added ranger policy
I20260810 11:39:32.066017 4035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms:/tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 11:39:32.066083 4035 mini_ranger_kms.cc:240] Using host: 127.3.240.212
I20260810 11:39:32.099850 4035 mini_ranger_kms.cc:276] Ranger KMS PID: 20480
I20260810 11:39:32.100059 4035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.240.212:42325
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:39:33 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 11:39:34 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786361974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:39:34.979941 4035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.864s user 0.168s sys 0.531s
I20260810 11:39:34.980332 4035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 11:39:34.980584 4035 mini_ranger_kms.cc:401] 127.3.240.212:42325/kms/v1/keys
Aug 10 11:39:35 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Aug 10 11:39:35 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361974, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Aug 10 11:39:38 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:39:38.247605 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45051
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45051
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42325/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:39:38.821252 20563 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:39:38.821593 20563 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:39:38.821858 20563 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:39:38.831686 20563 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:39:38.831929 20563 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:39:38.832057 20563 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:39:38.832192 20563 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:39:38.832430 20563 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:39:38.845091 20563 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.240.212:42325/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45051
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45051
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:39:38.846553 20563 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:39:38.848094 20563 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:39:38.869359 20569 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:38.869427 20570 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:38.872263 20575 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:39:38.873893 20563 server_base.cc:1034] running on GCE node
I20260810 11:39:38.875248 20563 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:39:38.877578 20563 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:39:38.878998 20563 hybrid_clock.cc:648] HybridClock initialized: now 1786361978878849 us; error 91 us; skew 500 ppm
I20260810 11:39:38.879670 20563 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:39:38 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:39:38.885172 20563 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:39:38.887250 20563 webserver.cc:460] Webserver started at http://127.3.240.254:36693/ using document root <none> and password file <none>
I20260810 11:39:38.888134 20563 fs_manager.cc:359] Metadata directory not provided
I20260810 11:39:38.888309 20563 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:39:38.888720 20563 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:39:38 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:38.957096 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "f0e5fea4303940bea7039c612dce27ac"
format_stamp: "Formatted at 2026-08-10 11:39:38 on dist-test-slave-drl0"
server_key: "ba7a9087c66b5ca91f47381168d19197"
server_key_iv: "ca92a1f2e724dca91e491f21eed8b38f"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:38.958122 20563 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f0e5fea4303940bea7039c612dce27ac"
format_stamp: "Formatted at 2026-08-10 11:39:38 on dist-test-slave-drl0"
server_key: "ba7a9087c66b5ca91f47381168d19197"
server_key_iv: "ca92a1f2e724dca91e491f21eed8b38f"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:38.965837 20563 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.001s
I20260810 11:39:38.985780 20591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:38.987620 20563 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:39:38.988113 20563 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "f0e5fea4303940bea7039c612dce27ac"
format_stamp: "Formatted at 2026-08-10 11:39:38 on dist-test-slave-drl0"
server_key: "ba7a9087c66b5ca91f47381168d19197"
server_key_iv: "ca92a1f2e724dca91e491f21eed8b38f"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:38.988489 20563 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:39:39.009697 20563 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:39:39.015576 20563 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:39:39.016002 20563 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:39:39.055786 20563 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45051
I20260810 11:39:39.055925 20664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45051 every 8 connection(s)
I20260810 11:39:39.057447 20563 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:39:39.063681 20665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:39:39.067260 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 20563
I20260810 11:39:39.067507 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:39:39.068338 4035 external_mini_cluster.cc:1428] Setting key 9050baadec417683356d123b42fbbbbd
I20260810 11:39:39.080855 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: Bootstrap starting.
I20260810 11:39:39.085409 20665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: Neither blocks nor log segments found. Creating new log.
Aug 10 11:39:39 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:39:39.087167 20665 log.cc:824] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: Log is configured to *not* fsync() on all Append() calls
I20260810 11:39:39.092731 20665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: No bootstrap required, opened a new log
I20260810 11:39:39.099931 20672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:39.070535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35758 (local address 127.3.240.254:45051)
0810 11:39:39.072234 (+ 1699us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:39.072246 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:39.072314 (+ 68us) server_negotiation.cc:407] Connection header received
0810 11:39:39.075285 (+ 2971us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:39.075310 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:39.075687 (+ 377us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:39.076047 (+ 360us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:39.078043 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:39.079760 (+ 1717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:39.081849 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:39.083027 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:39.083248 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:39.088845 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:39.088866 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:39.088888 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:39.088991 (+ 103us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:39.094200 (+ 5209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:39.095337 (+ 1137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:39.095348 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:39.095357 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:39.095677 (+ 320us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:39.096321 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:39.096332 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:39.096335 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:39.097013 (+ 678us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:39.097275 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:39.098204 (+ 929us) server_negotiation.cc:299] Negotiation successful
0810 11:39:39.098582 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1406,"thread_start_us":1084,"threads_started":1}
I20260810 11:39:39.105243 20665 raft_consensus.cc:348] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } }
I20260810 11:39:39.105868 20665 raft_consensus.cc:374] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:39:39.106042 20665 raft_consensus.cc:729] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f0e5fea4303940bea7039c612dce27ac, State: Initialized, Role: FOLLOWER
I20260810 11:39:39.106823 20665 consensus_queue.cc:260] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } }
I20260810 11:39:39.107396 20665 raft_consensus.cc:388] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:39:39.107609 20665 raft_consensus.cc:482] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:39:39.107786 20665 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:39:39.111285 20665 raft_consensus.cc:504] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } }
I20260810 11:39:39.112093 20665 leader_election.cc:302] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f0e5fea4303940bea7039c612dce27ac; no voters:
I20260810 11:39:39.113206 20665 leader_election.cc:288] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:39:39.113432 20675 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:39:39.115271 20675 raft_consensus.cc:686] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [term 1 LEADER]: Becoming Leader. State: Replica: f0e5fea4303940bea7039c612dce27ac, State: Running, Role: LEADER
I20260810 11:39:39.116093 20675 consensus_queue.cc:237] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } }
I20260810 11:39:39.116516 20665 sys_catalog.cc:564] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:39:39.120433 20677 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [sys.catalog]: SysCatalogTable state changed. Reason: New leader f0e5fea4303940bea7039c612dce27ac. Latest consensus state: current_term: 1 leader_uuid: "f0e5fea4303940bea7039c612dce27ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } } }
I20260810 11:39:39.121104 20676 sys_catalog.cc:455] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f0e5fea4303940bea7039c612dce27ac" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f0e5fea4303940bea7039c612dce27ac" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45051 } } }
I20260810 11:39:39.121951 20676 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [sys.catalog]: This master's current role is: LEADER
I20260810 11:39:39.121986 20677 sys_catalog.cc:458] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac [sys.catalog]: This master's current role is: LEADER
I20260810 11:39:39.133925 20685 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:39:39.143296 20685 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:39:39.144641 20665 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 11:39:39.180686 20685 catalog_manager.cc:1347] Generated new cluster ID: 57dd4142519d440782e9b6b8938b8a2e
I20260810 11:39:39.180932 20685 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:39:39.199517 20685 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:39:39.201300 20685 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:39:39.220836 20685 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: Generated new TSK 0
I20260810 11:39:39.222024 20685 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:39:40.204766 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:39:40 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 11:39:40.847994 20733 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:39:40.848301 20733 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:39:40.848457 20733 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:39:40.857349 20733 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:39:40.857584 20733 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:39:40.857897 20733 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:39:40.874795 20733 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.240.212:42325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:39:40.876981 20733 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:39:40.878720 20733 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:39:40.903726 20750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:40.904692 20751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:40.914848 20754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:39:40.915197 20733 server_base.cc:1034] running on GCE node
I20260810 11:39:40.915969 20733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:39:40.917236 20733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:39:40.918660 20733 hybrid_clock.cc:648] HybridClock initialized: now 1786361980918567 us; error 53 us; skew 500 ppm
I20260810 11:39:40.919309 20733 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:39:40 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:39:40.924245 20733 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:39:40.927162 20733 webserver.cc:460] Webserver started at http://127.3.240.193:42767/ using document root <none> and password file <none>
I20260810 11:39:40.928330 20733 fs_manager.cc:359] Metadata directory not provided
I20260810 11:39:40.928532 20733 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:39:40.929244 20733 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:39:40 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:40.953603 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "82f09e98b4f54ed09f101542a78e0f58"
format_stamp: "Formatted at 2026-08-10 11:39:40 on dist-test-slave-drl0"
server_key: "7588fb4d25d20ee4a30502ef8307607d"
server_key_iv: "f81ab2995d8ca7a966979a0209880ce9"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:40.954967 20733 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "82f09e98b4f54ed09f101542a78e0f58"
format_stamp: "Formatted at 2026-08-10 11:39:40 on dist-test-slave-drl0"
server_key: "7588fb4d25d20ee4a30502ef8307607d"
server_key_iv: "f81ab2995d8ca7a966979a0209880ce9"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:40.971442 20733 fs_manager.cc:691] Time spent creating directory manager: real 0.016s user 0.010s sys 0.004s
I20260810 11:39:41.005630 20769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:41.007910 20733 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.004s
I20260810 11:39:41.008285 20733 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "82f09e98b4f54ed09f101542a78e0f58"
format_stamp: "Formatted at 2026-08-10 11:39:40 on dist-test-slave-drl0"
server_key: "7588fb4d25d20ee4a30502ef8307607d"
server_key_iv: "f81ab2995d8ca7a966979a0209880ce9"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:41.008711 20733 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:39:41.061107 20733 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:39:41.068996 20733 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:39:41.069669 20733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:39:41.081799 20733 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:39:41.091711 20733 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:39:41.092028 20733 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:41.092482 20733 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:39:41.092785 20733 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:39:41 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:41.259635 20733 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37093
I20260810 11:39:41.259693 20937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37093 every 8 connection(s)
I20260810 11:39:41.261471 20733 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:39:41.269654 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 20733
I20260810 11:39:41.269909 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:39:41.271077 4035 external_mini_cluster.cc:1428] Setting key 5fa2d1670ff824ce892f28c5a92d4a57
Aug 10 11:39:41 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:39:41.311782 20944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:41.266616 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:44721 (local address 127.3.240.254:45051)
0810 11:39:41.279596 (+ 12980us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:41.279603 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:41.285139 (+ 5536us) server_negotiation.cc:407] Connection header received
0810 11:39:41.285224 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:41.285230 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:41.285325 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:41.285446 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:41.288343 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:41.289880 (+ 1537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:41.293055 (+ 3175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:41.293976 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:41.294077 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:41.301254 (+ 7177us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:41.301264 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:41.301270 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:41.301318 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:41.306649 (+ 5331us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:41.308046 (+ 1397us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:41.308052 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:41.308056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:41.308261 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:41.308859 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:41.308864 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:41.308867 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:41.309257 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:41.309393 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:41.311026 (+ 1633us) server_negotiation.cc:299] Negotiation successful
0810 11:39:41.311300 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12806,"thread_start_us":12613,"threads_started":1}
I20260810 11:39:41.314838 20943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:41.265762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45051 (local address 127.3.240.193:44721)
0810 11:39:41.280936 (+ 15174us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:41.281008 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:41.284690 (+ 3682us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:41.286070 (+ 1380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:41.286084 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:41.286846 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:39:41.288093 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:41.288125 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:41.290162 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:41.290169 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:41.292857 (+ 2688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:41.292874 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:41.294916 (+ 2042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:41.294925 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:41.295876 (+ 951us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:41.296910 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:39:41.296943 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:39:41.300876 (+ 3933us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:39:41.306882 (+ 6006us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:41.306893 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:41.306914 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:41.307882 (+ 968us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:41.308420 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:41.308425 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:41.308429 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:41.308720 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:41.309510 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:39:41.309519 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:39:41.309970 (+ 451us) client_negotiation.cc:769] Sending connection context
0810 11:39:41.312367 (+ 2397us) client_negotiation.cc:240] Negotiation successful
0810 11:39:41.313184 (+ 817us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14773,"thread_start_us":14221,"threads_started":1}
I20260810 11:39:41.317381 20940 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45051
I20260810 11:39:41.317809 20940 heartbeater.cc:464] Registering TS with master...
I20260810 11:39:41.318981 20940 heartbeater.cc:511] Master 127.3.240.254:45051 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:39:41.321770 20617 ts_manager.cc:194] Registered new tserver with Master: 82f09e98b4f54ed09f101542a78e0f58 (127.3.240.193:37093)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260810 11:39:41.324326 20617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:44721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:39:41.375571 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:39:41 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361980, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
W20260810 11:39:42.085867 20953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:39:42.086218 20953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:39:42.086467 20953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:39:42.097826 20953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:39:42.098109 20953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:39:42.098369 20953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:39:42.112526 20953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.240.212:42325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:39:42.114185 20953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:39:42.115816 20953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:39:42.137858 20977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:42.137827 20974 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:42.139745 20971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:39:42.145171 20953 server_base.cc:1034] running on GCE node
I20260810 11:39:42.145941 20953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:39:42.147161 20953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:39:42.148514 20953 hybrid_clock.cc:648] HybridClock initialized: now 1786361982148337 us; error 91 us; skew 500 ppm
I20260810 11:39:42.148921 20953 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:39:42 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361982, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:39:42.158670 20953 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:39:42.160985 20953 webserver.cc:460] Webserver started at http://127.3.240.194:38325/ using document root <none> and password file <none>
I20260810 11:39:42.162074 20953 fs_manager.cc:359] Metadata directory not provided
I20260810 11:39:42.162257 20953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:39:42.162645 20953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:39:42 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361982, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:42.183743 20953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "c70ea2fd116743c2838e407019b86688"
format_stamp: "Formatted at 2026-08-10 11:39:42 on dist-test-slave-drl0"
server_key: "cf85b057cadab8e6316fc99d46d50378"
server_key_iv: "19a523793fb97c366f0fed03f94ad8f6"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:42.185050 20953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "c70ea2fd116743c2838e407019b86688"
format_stamp: "Formatted at 2026-08-10 11:39:42 on dist-test-slave-drl0"
server_key: "cf85b057cadab8e6316fc99d46d50378"
server_key_iv: "19a523793fb97c366f0fed03f94ad8f6"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:42.194586 20953 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.001s sys 0.009s
I20260810 11:39:42.218361 20988 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:42.220350 20953 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 11:39:42.220630 20953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "c70ea2fd116743c2838e407019b86688"
format_stamp: "Formatted at 2026-08-10 11:39:42 on dist-test-slave-drl0"
server_key: "cf85b057cadab8e6316fc99d46d50378"
server_key_iv: "19a523793fb97c366f0fed03f94ad8f6"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:42.221297 20953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:39:42.256217 20953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:39:42.262392 20953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:39:42.263087 20953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:39:42.267036 20953 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:39:42.269306 20953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:39:42.269414 20953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:42.269579 20953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:39:42.269668 20953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:42.328923 20940 heartbeater.cc:503] Master 127.3.240.254:45051 was elected leader, sending a full tablet report...
I20260810 11:39:42.349602 20953 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:41295
I20260810 11:39:42.349689 21163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:41295 every 8 connection(s)
I20260810 11:39:42.352710 20953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:39:42.357276 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 20953
I20260810 11:39:42.357589 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:39:42.358464 4035 external_mini_cluster.cc:1428] Setting key e5af9a7de0f092cc1b45e3b76cff2952
Aug 10 11:39:42 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361982, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:39:42.400244 21168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:42.356927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:37691 (local address 127.3.240.254:45051)
0810 11:39:42.369287 (+ 12360us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:42.369298 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:42.370994 (+ 1696us) server_negotiation.cc:407] Connection header received
0810 11:39:42.374578 (+ 3584us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:42.374584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:42.374675 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:42.374808 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:42.377912 (+ 3104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:42.379649 (+ 1737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:42.382967 (+ 3318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:42.383851 (+ 884us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:42.384002 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:42.389923 (+ 5921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:42.389933 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:42.389942 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:42.390017 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:42.395555 (+ 5538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:42.396659 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:42.396667 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:42.396673 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:42.396885 (+ 212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:42.397577 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:42.397583 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:42.397586 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:42.398050 (+ 464us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:42.398227 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:42.399296 (+ 1069us) server_negotiation.cc:299] Negotiation successful
0810 11:39:42.399560 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12163,"thread_start_us":12368,"threads_started":1}
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
I20260810 11:39:42.401695 21169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:42.358229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45051 (local address 127.3.240.194:37691)
0810 11:39:42.370417 (+ 12188us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:42.370483 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:42.374206 (+ 3723us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:42.375162 (+ 956us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:42.375178 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:42.376263 (+ 1085us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:39:42.377666 (+ 1403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:42.377707 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:42.379844 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:42.379853 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:42.382737 (+ 2884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:42.382759 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:42.384160 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:42.384171 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:42.384688 (+ 517us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:42.385824 (+ 1136us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:39:42.385863 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:39:42.389568 (+ 3705us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:39:42.395760 (+ 6192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:42.395771 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:42.395793 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:42.396488 (+ 695us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:42.397042 (+ 554us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:42.397047 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:42.397051 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:42.397412 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:42.398224 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:39:42.398233 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:39:42.398706 (+ 473us) client_negotiation.cc:769] Sending connection context
0810 11:39:42.399107 (+ 401us) client_negotiation.cc:240] Negotiation successful
0810 11:39:42.400019 (+ 912us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11774,"thread_start_us":11458,"threads_started":1}
I20260810 11:39:42.403540 21165 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45051
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
I20260810 11:39:42.404026 21165 heartbeater.cc:464] Registering TS with master...
I20260810 11:39:42.405304 21165 heartbeater.cc:511] Master 127.3.240.254:45051 requested a full tablet report, sending...
I20260810 11:39:42.407444 20617 ts_manager.cc:194] Registered new tserver with Master: c70ea2fd116743c2838e407019b86688 (127.3.240.194:41295)
I20260810 11:39:42.408632 20617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:37691
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:39:42.446383 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42325/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:46213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:39:43.057109 21173 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:39:43.057453 21173 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:39:43.057646 21173 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:39:43.067180 21173 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:39:43.067375 21173 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:39:43.067634 21173 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:39:43.081876 21173 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:46213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.240.212:42325/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:45051
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:39:43.083460 21173 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:39:43.084973 21173 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:39:43.103858 21181 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:43.103812 21182 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:39:43.103910 21185 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:39:43.105476 21173 server_base.cc:1034] running on GCE node
I20260810 11:39:43.106387 21173 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:39:43.107818 21173 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:39:43.109180 21173 hybrid_clock.cc:648] HybridClock initialized: now 1786361983109037 us; error 54 us; skew 500 ppm
I20260810 11:39:43.109685 21173 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:39:43 dist-test-slave-drl0 krb5kdc[19891](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361983, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:39:43.115373 21173 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:39:43.117857 21173 webserver.cc:460] Webserver started at http://127.3.240.195:39173/ using document root <none> and password file <none>
I20260810 11:39:43.118764 21173 fs_manager.cc:359] Metadata directory not provided
I20260810 11:39:43.118991 21173 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:39:43.119414 21173 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:39:43 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361983, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:39:43.139730 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "b70ce7458d0f4229b0a7425846659e7c"
format_stamp: "Formatted at 2026-08-10 11:39:43 on dist-test-slave-drl0"
server_key: "d6ccc3e885062c465fb4ee053a1a5271"
server_key_iv: "e1f8a65fdff5c5d02e88369f2f138641"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:43.140854 21173 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "b70ce7458d0f4229b0a7425846659e7c"
format_stamp: "Formatted at 2026-08-10 11:39:43 on dist-test-slave-drl0"
server_key: "d6ccc3e885062c465fb4ee053a1a5271"
server_key_iv: "e1f8a65fdff5c5d02e88369f2f138641"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:43.149346 21173 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.004s sys 0.004s
I20260810 11:39:43.167647 21198 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:43.170246 21173 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260810 11:39:43.170537 21173 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "b70ce7458d0f4229b0a7425846659e7c"
format_stamp: "Formatted at 2026-08-10 11:39:43 on dist-test-slave-drl0"
server_key: "d6ccc3e885062c465fb4ee053a1a5271"
server_key_iv: "e1f8a65fdff5c5d02e88369f2f138641"
server_key_version: "kuduclusterkey@0"
I20260810 11:39:43.170892 21173 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:39:43.209431 21173 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:39:43.214265 21173 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:39:43.214608 21173 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:39:43.216681 21173 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:39:43.218787 21173 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:39:43.218875 21173 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:43.219012 21173 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:39:43.219071 21173 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:39:43.289606 21173 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34653
I20260810 11:39:43.289628 21367 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34653 every 8 connection(s)
I20260810 11:39:43.291086 21173 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:39:43.299324 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 21173
I20260810 11:39:43.299556 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:39:43.300117 4035 external_mini_cluster.cc:1428] Setting key fce6e9c2af2c066c759ec42f1030785b
Aug 10 11:39:43 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361983, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:39:43.319581 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.294289 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:54133 (local address 127.3.240.254:45051)
0810 11:39:43.295047 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.295052 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.297028 (+ 1976us) server_negotiation.cc:407] Connection header received
0810 11:39:43.299425 (+ 2397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.299431 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.299556 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.299704 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:43.301841 (+ 2137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.303213 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.305467 (+ 2254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.306343 (+ 876us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.306453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.311101 (+ 4648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:43.311111 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:43.311117 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:43.311150 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:43.314488 (+ 3338us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.316145 (+ 1657us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.316151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.316155 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.316594 (+ 439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.317444 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.317450 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.317453 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.318021 (+ 568us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:43.318180 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.319046 (+ 866us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.319260 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":405,"threads_started":1}
I20260810 11:39:43.320457 21371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.295525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45051 (local address 127.3.240.195:54133)
0810 11:39:43.296856 (+ 1331us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.296908 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.299123 (+ 2215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.299961 (+ 838us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.299969 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.300600 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:39:43.301594 (+ 994us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.301638 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.303366 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.303371 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.305243 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.305256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.306651 (+ 1395us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.306658 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.307040 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.307944 (+ 904us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:39:43.307968 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:39:43.310531 (+ 2563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:39:43.314779 (+ 4248us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:43.314797 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:43.314832 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:43.315982 (+ 1150us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:43.316758 (+ 776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:39:43.316764 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:39:43.316767 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:39:43.317089 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:39:43.318168 (+ 1079us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:39:43.318174 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:39:43.318487 (+ 313us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.318729 (+ 242us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.319239 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1082,"thread_start_us":681,"threads_started":1}
I20260810 11:39:43.322053 21368 heartbeater.cc:347] Connected to a master server at 127.3.240.254:45051
I20260810 11:39:43.322402 21368 heartbeater.cc:464] Registering TS with master...
I20260810 11:39:43.323321 21368 heartbeater.cc:511] Master 127.3.240.254:45051 requested a full tablet report, sending...
I20260810 11:39:43.325006 20617 ts_manager.cc:194] Registered new tserver with Master: b70ce7458d0f4229b0a7425846659e7c (127.3.240.195:34653)
I20260810 11:39:43.325999 20617 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:54133
I20260810 11:39:43.330338 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:39:43.356021 21370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.337852 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.3.240.254:45051)
0810 11:39:43.338190 (+ 338us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.338212 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.338812 (+ 600us) server_negotiation.cc:407] Connection header received
0810 11:39:43.339079 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.339084 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.339174 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.339295 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:43.341265 (+ 1970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.342061 (+ 796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.343802 (+ 1741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.344344 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.344504 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.347758 (+ 3254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:43.347768 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:43.347773 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:43.347845 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:43.351812 (+ 3967us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.353314 (+ 1502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.353320 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.353322 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.353485 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.353998 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.354004 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.354007 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.354310 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:43.354591 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.355218 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.355635 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260810 11:39:43.361545 20617 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53440:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:39:43.372118 20617 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:39:43.405714 21392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.389154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37093 (local address 127.0.0.1:36432)
0810 11:39:43.391854 (+ 2700us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.391927 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.392187 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.394706 (+ 2519us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.394711 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.394743 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:39:43.395887 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.395902 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.398853 (+ 2951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.398860 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.402435 (+ 3575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.402448 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.404541 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.404553 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.404766 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.404822 (+ 56us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.404939 (+ 117us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.405139 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2137,"mutex_wait_us":39,"thread_start_us":1844,"threads_started":1}
I20260810 11:39:43.406190 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.389799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36432 (local address 127.3.240.193:37093)
0810 11:39:43.391467 (+ 1668us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.391479 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.392441 (+ 962us) server_negotiation.cc:407] Connection header received
0810 11:39:43.392655 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.392666 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.394001 (+ 1335us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.394242 (+ 241us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:39:43.396076 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.398719 (+ 2643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.402717 (+ 3998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.404181 (+ 1464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.404374 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.405245 (+ 871us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.405355 (+ 110us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.405538 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1308,"thread_start_us":924,"threads_started":1}
I20260810 11:39:43.406757 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.389421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50874 (local address 127.3.240.194:41295)
0810 11:39:43.390363 (+ 942us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.390374 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.392549 (+ 2175us) server_negotiation.cc:407] Connection header received
0810 11:39:43.392767 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.392791 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.393182 (+ 391us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.393461 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:39:43.395317 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.397875 (+ 2558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.403052 (+ 5177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.404465 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.404610 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.404871 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.406155 (+ 1284us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.406391 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":656,"thread_start_us":481,"threads_started":1}
I20260810 11:39:43.407272 21395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.392346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34653 (local address 127.0.0.1:56508)
0810 11:39:43.395145 (+ 2799us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.395168 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.395342 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.396419 (+ 1077us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.396424 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.396454 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:39:43.397979 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.397995 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.400884 (+ 2889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.400890 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.403665 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.403680 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.406209 (+ 2529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.406219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.406428 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.406464 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.406584 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.406744 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2627,"mutex_wait_us":1,"thread_start_us":1000,"threads_started":1}
I20260810 11:39:43.407713 21393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.390277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41295 (local address 127.0.0.1:50874)
0810 11:39:43.392447 (+ 2170us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.392466 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.392623 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.393519 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.393527 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.393558 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:39:43.394784 (+ 1226us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.394806 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.398090 (+ 3284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.398094 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.402805 (+ 4711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.402820 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.405722 (+ 2902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.405731 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.405942 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.405973 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.407002 (+ 1029us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.407139 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1960,"thread_start_us":1638,"threads_started":1}
I20260810 11:39:43.408264 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.393365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56508 (local address 127.3.240.195:34653)
0810 11:39:43.395194 (+ 1829us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.395206 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.395268 (+ 62us) server_negotiation.cc:407] Connection header received
0810 11:39:43.395587 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.395595 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.395903 (+ 308us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.396309 (+ 406us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:39:43.398420 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.400615 (+ 2195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.403880 (+ 3265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.405538 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.405671 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.405982 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.407503 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.407689 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1462,"thread_start_us":1113,"threads_started":1}
I20260810 11:39:43.411124 20843 tablet_service.cc:1467] Processing CreateTablet for tablet ebb2531af787407a99d5d520a85f32bb (DEFAULT_TABLE table=test-table [id=c131fff15c6d4e78a13450a53d47e33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:39:43.411310 21054 tablet_service.cc:1467] Processing CreateTablet for tablet ebb2531af787407a99d5d520a85f32bb (DEFAULT_TABLE table=test-table [id=c131fff15c6d4e78a13450a53d47e33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:39:43.411757 21271 tablet_service.cc:1467] Processing CreateTablet for tablet ebb2531af787407a99d5d520a85f32bb (DEFAULT_TABLE table=test-table [id=c131fff15c6d4e78a13450a53d47e33a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:39:43.412623 21165 heartbeater.cc:503] Master 127.3.240.254:45051 was elected leader, sending a full tablet report...
I20260810 11:39:43.413306 20843 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb2531af787407a99d5d520a85f32bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:39:43.413344 21054 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb2531af787407a99d5d520a85f32bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:39:43.413784 21271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ebb2531af787407a99d5d520a85f32bb. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:39:43.431937 21397 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Bootstrap starting.
I20260810 11:39:43.434427 21398 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Bootstrap starting.
I20260810 11:39:43.435760 21399 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Bootstrap starting.
I20260810 11:39:43.437543 21397 tablet_bootstrap.cc:654] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Neither blocks nor log segments found. Creating new log.
I20260810 11:39:43.439504 21397 log.cc:824] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Log is configured to *not* fsync() on all Append() calls
I20260810 11:39:43.440534 21398 tablet_bootstrap.cc:654] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Neither blocks nor log segments found. Creating new log.
I20260810 11:39:43.442368 21398 log.cc:824] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:39:43.443542 21397 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: No bootstrap required, opened a new log
I20260810 11:39:43.443964 21397 ts_tablet_manager.cc:1397] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.009s
I20260810 11:39:43.445289 21399 tablet_bootstrap.cc:654] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Neither blocks nor log segments found. Creating new log.
I20260810 11:39:43.446770 21398 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: No bootstrap required, opened a new log
I20260810 11:39:43.447113 21398 ts_tablet_manager.cc:1397] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.004s
I20260810 11:39:43.447119 21399 log.cc:824] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Log is configured to *not* fsync() on all Append() calls
I20260810 11:39:43.451099 21399 tablet_bootstrap.cc:492] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: No bootstrap required, opened a new log
I20260810 11:39:43.451462 21399 ts_tablet_manager.cc:1397] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260810 11:39:43.453732 21397 raft_consensus.cc:348] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.454146 21397 raft_consensus.cc:374] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:39:43.454299 21397 raft_consensus.cc:729] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 82f09e98b4f54ed09f101542a78e0f58, State: Initialized, Role: FOLLOWER
I20260810 11:39:43.455061 21397 consensus_queue.cc:260] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.457798 21397 ts_tablet_manager.cc:1428] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Time spent starting tablet: real 0.014s user 0.010s sys 0.003s
I20260810 11:39:43.462328 21398 raft_consensus.cc:348] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.463142 21398 raft_consensus.cc:374] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:39:43.463378 21398 raft_consensus.cc:729] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b70ce7458d0f4229b0a7425846659e7c, State: Initialized, Role: FOLLOWER
I20260810 11:39:43.464183 21398 consensus_queue.cc:260] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.465653 21368 heartbeater.cc:503] Master 127.3.240.254:45051 was elected leader, sending a full tablet report...
I20260810 11:39:43.466502 21398 ts_tablet_manager.cc:1428] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Time spent starting tablet: real 0.019s user 0.009s sys 0.009s
I20260810 11:39:43.466216 21399 raft_consensus.cc:348] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.466699 21399 raft_consensus.cc:374] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:39:43.466908 21399 raft_consensus.cc:729] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c70ea2fd116743c2838e407019b86688, State: Initialized, Role: FOLLOWER
I20260810 11:39:43.467590 21399 consensus_queue.cc:260] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.469744 21399 ts_tablet_manager.cc:1428] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260810 11:39:43.528856 20941 tablet.cc:2369] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:39:43.543253 21369 tablet.cc:2369] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:39:43.606648 21167 tablet.cc:2369] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:39:43.690800 21408 raft_consensus.cc:482] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:39:43.691272 21408 raft_consensus.cc:504] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.694640 21408 leader_election.cc:288] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c70ea2fd116743c2838e407019b86688 (127.3.240.194:41295), b70ce7458d0f4229b0a7425846659e7c (127.3.240.195:34653)
I20260810 11:39:43.705349 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.695130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56533 (local address 127.3.240.194:41295)
0810 11:39:43.696157 (+ 1027us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.696163 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.697773 (+ 1610us) server_negotiation.cc:407] Connection header received
0810 11:39:43.698026 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.698030 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.698092 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.698188 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:39:43.699255 (+ 1067us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.700353 (+ 1098us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.702869 (+ 2516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.704349 (+ 1480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.704561 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.704651 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.704859 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.705041 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":815}
I20260810 11:39:43.705397 21412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.694832 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41295 (local address 127.3.240.193:56533)
0810 11:39:43.697636 (+ 2804us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.697660 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.697872 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.698227 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.698232 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.698263 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:39:43.699075 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.699091 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.700533 (+ 1442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.700539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.702687 (+ 2148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.702700 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.704569 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.704574 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.704699 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.704724 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.704806 (+ 82us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.704893 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2466,"mutex_wait_us":4,"thread_start_us":3306,"threads_started":1}
I20260810 11:39:43.706385 21411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.695441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34653 (local address 127.3.240.193:44575)
0810 11:39:43.698685 (+ 3244us) negotiation.cc:107] Waiting for socket to connect
0810 11:39:43.698703 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:39:43.698832 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:39:43.699196 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:39:43.699201 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:39:43.699227 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:39:43.699691 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.699701 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.701466 (+ 1765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.701471 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.703335 (+ 1864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:39:43.703348 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.705402 (+ 2054us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.705411 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:39:43.705839 (+ 428us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.705883 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 11:39:43.705997 (+ 114us) client_negotiation.cc:240] Negotiation successful
0810 11:39:43.706134 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2929,"thread_start_us":1322,"threads_started":1}
I20260810 11:39:43.706735 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.695415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:44575 (local address 127.3.240.195:34653)
0810 11:39:43.695658 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.695665 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.698810 (+ 3145us) server_negotiation.cc:407] Connection header received
0810 11:39:43.698973 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.698978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.699062 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.699165 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:39:43.699856 (+ 691us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.701202 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.703513 (+ 2311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.705110 (+ 1597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.705289 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.705370 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.706032 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.706188 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 11:39:43.706826 21089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb2531af787407a99d5d520a85f32bb" candidate_uuid: "82f09e98b4f54ed09f101542a78e0f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70ea2fd116743c2838e407019b86688" is_pre_election: true
I20260810 11:39:43.707541 21089 raft_consensus.cc:2454] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f09e98b4f54ed09f101542a78e0f58 in term 0.
I20260810 11:39:43.708122 21299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb2531af787407a99d5d520a85f32bb" candidate_uuid: "82f09e98b4f54ed09f101542a78e0f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70ce7458d0f4229b0a7425846659e7c" is_pre_election: true
I20260810 11:39:43.708448 20778 leader_election.cc:302] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f09e98b4f54ed09f101542a78e0f58, c70ea2fd116743c2838e407019b86688; no voters:
I20260810 11:39:43.708794 21299 raft_consensus.cc:2454] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 82f09e98b4f54ed09f101542a78e0f58 in term 0.
I20260810 11:39:43.709277 21408 raft_consensus.cc:2781] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:39:43.709744 21408 raft_consensus.cc:482] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:39:43.709996 21408 raft_consensus.cc:3037] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:39:43.712610 21408 raft_consensus.cc:504] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.713299 21408 leader_election.cc:288] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [CANDIDATE]: Term 1 election: Requested vote from peers c70ea2fd116743c2838e407019b86688 (127.3.240.194:41295), b70ce7458d0f4229b0a7425846659e7c (127.3.240.195:34653)
I20260810 11:39:43.713776 21089 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb2531af787407a99d5d520a85f32bb" candidate_uuid: "82f09e98b4f54ed09f101542a78e0f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c70ea2fd116743c2838e407019b86688"
I20260810 11:39:43.713930 21299 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ebb2531af787407a99d5d520a85f32bb" candidate_uuid: "82f09e98b4f54ed09f101542a78e0f58" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b70ce7458d0f4229b0a7425846659e7c"
I20260810 11:39:43.714044 21089 raft_consensus.cc:3037] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:39:43.714171 21299 raft_consensus.cc:3037] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:39:43.717243 21299 raft_consensus.cc:2454] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f09e98b4f54ed09f101542a78e0f58 in term 1.
I20260810 11:39:43.717535 21089 raft_consensus.cc:2454] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 82f09e98b4f54ed09f101542a78e0f58 in term 1.
I20260810 11:39:43.717716 20775 leader_election.cc:302] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 82f09e98b4f54ed09f101542a78e0f58, b70ce7458d0f4229b0a7425846659e7c; no voters:
I20260810 11:39:43.718163 21408 raft_consensus.cc:2781] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:39:43.719240 21408 raft_consensus.cc:686] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 1 LEADER]: Becoming Leader. State: Replica: 82f09e98b4f54ed09f101542a78e0f58, State: Running, Role: LEADER
I20260810 11:39:43.719892 21408 consensus_queue.cc:237] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } }
I20260810 11:39:43.725148 20616 catalog_manager.cc:5495] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 reported cstate change: term changed from 0 to 1, leader changed from <none> to 82f09e98b4f54ed09f101542a78e0f58 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "82f09e98b4f54ed09f101542a78e0f58" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "82f09e98b4f54ed09f101542a78e0f58" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37093 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 } health_report { overall_health: UNKNOWN } } }
I20260810 11:39:43.783893 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.777430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36434 (local address 127.3.240.193:37093)
0810 11:39:43.777770 (+ 340us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.777786 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.777807 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:39:43.778075 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.778080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.778174 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.778337 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:39:43.778751 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.779659 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.781281 (+ 1622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.781837 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.781982 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.782613 (+ 631us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:39:43.783241 (+ 628us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:39:43.783453 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.783506 (+ 53us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.783587 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 11:39:43.795696 21089 raft_consensus.cc:1261] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 1 FOLLOWER]: Refusing update from remote peer 82f09e98b4f54ed09f101542a78e0f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:39:43.795722 21299 raft_consensus.cc:1261] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 1 FOLLOWER]: Refusing update from remote peer 82f09e98b4f54ed09f101542a78e0f58: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:39:43.796757 21413 consensus_queue.cc:1035] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b70ce7458d0f4229b0a7425846659e7c" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:39:43.797207 21408 consensus_queue.cc:1035] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c70ea2fd116743c2838e407019b86688" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41295 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:39:43.810408 21422 mvcc.cc:205] Tried to move back new op lower bound from 7316938685613580288 to 7316938685320724480. Current Snapshot: MvccSnapshot[applied={T|T < 7316938685613580288}]
I20260810 11:39:43.830209 20616 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:39:43.830541 20616 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53440:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:39:43.835446 20616 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f0e5fea4303940bea7039c612dce27ac: Sending DeleteTablet for 3 replicas of tablet ebb2531af787407a99d5d520a85f32bb
I20260810 11:39:43.836822 21054 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb2531af787407a99d5d520a85f32bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:39:43 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:50874
I20260810 11:39:43.837061 21271 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb2531af787407a99d5d520a85f32bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:39:43 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:56508
I20260810 11:39:43.837699 20843 tablet_service.cc:1514] Processing DeleteTablet for tablet ebb2531af787407a99d5d520a85f32bb with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:39:43 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:36432
I20260810 11:39:43.839387 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 20733
I20260810 11:39:43.847862 21436 tablet_replica.cc:330] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: stopping tablet replica
I20260810 11:39:43.849233 21436 raft_consensus.cc:2229] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:39:43.850709 21436 raft_consensus.cc:2258] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:39:43.851562 21437 tablet_replica.cc:330] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: stopping tablet replica
I20260810 11:39:43.852633 21437 raft_consensus.cc:2229] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:39:43.853586 21439 tablet_replica.cc:330] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: stopping tablet replica
I20260810 11:39:43.854435 21439 raft_consensus.cc:2229] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:39:43.855400 21437 raft_consensus.cc:2258] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:39:43.855600 21439 raft_consensus.cc:2258] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:39:43.855854 21436 ts_tablet_manager.cc:1905] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:39:43.859112 21439 ts_tablet_manager.cc:1905] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 11:39:43 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:39:43.865218 21436 ts_tablet_manager.cc:1918] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:39:43.865592 21436 log.cc:1196] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/ebb2531af787407a99d5d520a85f32bb
I20260810 11:39:43.866668 21436 ts_tablet_manager.cc:1939] T ebb2531af787407a99d5d520a85f32bb P c70ea2fd116743c2838e407019b86688: Deleting consensus metadata
I20260810 11:39:43.866909 21437 ts_tablet_manager.cc:1905] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:39:43.869597 20596 catalog_manager.cc:4841] TS c70ea2fd116743c2838e407019b86688 (127.3.240.194:41295): tablet ebb2531af787407a99d5d520a85f32bb (table test-table [id=c131fff15c6d4e78a13450a53d47e33a]) successfully deleted
I20260810 11:39:43.872045 21439 ts_tablet_manager.cc:1918] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:39:43.872481 21439 log.cc:1196] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/ebb2531af787407a99d5d520a85f32bb
I20260810 11:39:43.874624 21439 ts_tablet_manager.cc:1939] T ebb2531af787407a99d5d520a85f32bb P 82f09e98b4f54ed09f101542a78e0f58: Deleting consensus metadata
I20260810 11:39:43.875312 21437 ts_tablet_manager.cc:1918] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:39:43.875576 21391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:43.840746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36448 (local address 127.3.240.193:37093)
0810 11:39:43.840923 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:43.840930 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:43.850800 (+ 9870us) server_negotiation.cc:407] Connection header received
0810 11:39:43.851298 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:43.851319 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:43.851413 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:43.851490 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:43.853707 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.854795 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.857972 (+ 3177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:43.858700 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:43.858845 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:43.864854 (+ 6009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:43.864869 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:43.864888 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:43.864935 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:43.870152 (+ 5217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.871165 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.871173 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.871177 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.871492 (+ 315us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:43.872222 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:43.872230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:43.872234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:43.873560 (+ 1326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:43.874226 (+ 666us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:43.874442 (+ 216us) server_negotiation.cc:299] Negotiation successful
0810 11:39:43.875085 (+ 643us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 11:39:43.875942 21437 log.cc:1196] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/ebb2531af787407a99d5d520a85f32bb
I20260810 11:39:43.876940 20932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:39:43.877039 20592 catalog_manager.cc:4841] TS 82f09e98b4f54ed09f101542a78e0f58 (127.3.240.193:37093): tablet ebb2531af787407a99d5d520a85f32bb (table test-table [id=c131fff15c6d4e78a13450a53d47e33a]) successfully deleted
I20260810 11:39:43.881968 21437 ts_tablet_manager.cc:1939] T ebb2531af787407a99d5d520a85f32bb P b70ce7458d0f4229b0a7425846659e7c: Deleting consensus metadata
I20260810 11:39:43.883613 20592 catalog_manager.cc:4841] TS b70ce7458d0f4229b0a7425846659e7c (127.3.240.195:34653): tablet ebb2531af787407a99d5d520a85f32bb (table test-table [id=c131fff15c6d4e78a13450a53d47e33a]) successfully deleted
I20260810 11:39:43.982400 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 20733
I20260810 11:39:44.009675 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 20953
Aug 10 11:39:44 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:39:44.028420 21390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:44.010834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50882 (local address 127.3.240.194:41295)
0810 11:39:44.011114 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:44.011118 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:44.011218 (+ 100us) server_negotiation.cc:407] Connection header received
0810 11:39:44.011484 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:44.011488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:44.011559 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:44.011651 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:44.013087 (+ 1436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:44.013771 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:44.015342 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:44.015746 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:44.015858 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:44.021151 (+ 5293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:44.021165 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:44.021182 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:44.021217 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:44.024382 (+ 3165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:44.025522 (+ 1140us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:44.025529 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:44.025531 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:44.025670 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:44.026219 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:44.026224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:44.026227 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:44.026831 (+ 604us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:44.027391 (+ 560us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:44.027687 (+ 296us) server_negotiation.cc:299] Negotiation successful
0810 11:39:44.027994 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260810 11:39:44.029565 21157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:39:44.123490 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 20953
I20260810 11:39:44.148947 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 21173
Aug 10 11:39:44 dist-test-slave-drl0 krb5kdc[19891](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361978, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:39:44.166535 21396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:39:44.149765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56520 (local address 127.3.240.195:34653)
0810 11:39:44.149965 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:39:44.149969 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:39:44.149988 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:39:44.150137 (+ 149us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:39:44.150141 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:39:44.150221 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:39:44.150314 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:39:44.151958 (+ 1644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:44.152968 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:44.154694 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:39:44.155351 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:39:44.155479 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:39:44.160518 (+ 5039us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:39:44.160533 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:39:44.160551 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:39:44.160586 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:39:44.163529 (+ 2943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:44.164266 (+ 737us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:44.164274 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:44.164276 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:44.164433 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:39:44.164834 (+ 401us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:39:44.164838 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:39:44.164840 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:39:44.165373 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:39:44.165590 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 11:39:44.166005 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 11:39:44.166261 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 11:39:44.167265 21360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:39:44.261210 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 21173
I20260810 11:39:44.287348 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 20563
I20260810 11:39:44.288831 20660 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:39:44.409433 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 20563
I20260810 11:39:44.446267 4035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 11:39:44.792759 4035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 11:39:44.793103 4035 mini_ranger.cc:66] Stopping Ranger...
I20260810 11:39:45.139555 4035 mini_ranger.cc:68] Stopped Ranger
2026-08-10 11:39:45.140 UTC [19936] LOG: received smart shutdown request
2026-08-10 11:39:45.144 UTC [19936] LOG: background worker "logical replication launcher" (PID 19952) exited with exit code 1
2026-08-10 11:39:45.145 UTC [19944] LOG: shutting down
2026-08-10 11:39:45.306 UTC [19936] LOG: database system is shut down
2026-08-10T11:39:45Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58160 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:39:45 dist-test-slave-drl0 krb5kdc[21465](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:39:45 dist-test-slave-drl0 krb5kdc[21465](info): set up 2 sockets
Aug 10 11:39:45 dist-test-slave-drl0 krb5kdc[21465](info): commencing operation
krb5kdc: starting...
W20260810 11:39:46.826016 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.003s sys 0.043s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:39:47 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786361987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:39:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:39:47Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangeradmin_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangerlookup_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
I20260810 11:39:47.333882 4035 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskagqWOB/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/postgres -l logfile start
2026-08-10 11:39:50.194 UTC [21513] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-10 11:39:50.194 UTC [21513] LOG: listening on IPv4 address "127.3.240.212", port 51159
2026-08-10 11:39:50.198 UTC [21513] LOG: listening on Unix socket "/tmp/.s.PGSQL.51159"
2026-08-10 11:39:50.223 UTC [21516] LOG: database system was shut down at 2026-08-10 11:39:48 UTC
2026-08-10 11:39:50.228 UTC [21513] LOG: database system is ready to accept connections
I20260810 11:39:51.457314 4035 mini_postgres.cc:96] Postgres bound to 51159
2026-08-10 11:39:51.485 UTC [21528] FATAL: database "slave" does not exist
127.3.240.212:51159 - accepting connections
I20260810 11:39:51.486846 4035 mini_ranger.cc:161] Starting Ranger...
I20260810 11:39:51.522639 4035 mini_ranger.cc:84] Created miniranger Postgres user
I20260810 11:39:52.116047 4035 mini_ranger.cc:87] Created ranger Postgres database
I20260810 11:39:52.116197 4035 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin
2026-08-10 11:39:52,451 [I] DB FLAVOR :POSTGRES
2026-08-10 11:39:52,451 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 11:39:52,451 [I] Checking connection..
2026-08-10 11:39:52,451 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-10 11:39:52,731 [I] Checking connection passed.
2026-08-10 11:39:52,732 [I] --------- Verifying version history table ---------
2026-08-10 11:39:52,732 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:39:53,018 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 11:39:53,018 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:39:53,315 [I] Table x_db_version_h does not exist in database ranger
2026-08-10 11:39:53,316 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-10 11:39:53,316 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-10 11:39:53.584 UTC [21625] WARNING: there is no transaction in progress
2026-08-10 11:39:53,595 [I] create_dbversion_catalog.sql file imported successfully
2026-08-10 11:39:53,595 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-10 11:39:53,883 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-10 11:39:53,884 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 11:39:53,884 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-10 11:39:54,176 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-10 11:39:54,483 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-drl0.c.gcp-upstream.internal','N') ;"
2026-08-10 11:39:54,750 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 11:39:55,027 [I] Table x_portal_user does not exist in database ranger
2026-08-10 11:39:55,027 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 11:39:55,319 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-10 11:39:55,319 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 11:39:55,598 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 11:39:55,878 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-10 11:39:55,878 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-10 11:39:57.016 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.035 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.054 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.073 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.094 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.118 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.191 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.198 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.216 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.227 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.239 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.256 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.268 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.276 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.285 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:57.293 UTC [21789] WARNING: there is no transaction in progress
2026-08-10 11:39:58,084 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-10 11:39:58,085 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-10 11:39:58,390 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-10 11:39:58,390 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-10 11:39:58,700 [I] Table x_portal_user already exists in database 'ranger'
2026-08-10 11:39:58,701 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-10 11:39:58,965 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-10 11:39:58,966 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-10 11:39:59,258 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-10 11:39:59,543 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-drl0.c.gcp-upstream.internal';"
2026-08-10 11:39:59,833 [I] CORE_DB_SCHEMA import status has been updated
2026-08-10 11:39:59,834 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-10 11:40:00,126 [I] DB_PATCHES have already been applied
I20260810 11:40:00.133255 4035 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskagqWOB/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260810 11:40:00.133389 4035 mini_ranger.cc:193] Using host: 127.3.240.212
I20260810 11:40:00.158253 4035 mini_ranger.cc:227] Ranger admin URL: http://127.3.240.212:60823
Aug 10, 2026 11:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp
Aug 10, 2026 11:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 10, 2026 11:40:00 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 10, 2026 11:40:01 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-admin/ews/webapp].
Aug 10, 2026 11:40:01 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260810 11:40:01.616236 4035 mini_ranger.cc:160] Time spent starting Ranger: real 10.129s user 0.003s sys 0.072s
Aug 10, 2026 11:40:22 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 10, 2026 11:40:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 10, 2026 11:40:22 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 10, 2026 11:40:22 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 10, 2026 11:40:23 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260810 11:40:24.292073 4035 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.3.240.212@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.3.240.212@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.3.240.212@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/rangerkms_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.3.240.212@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.3.240.212@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
Entry for principal HTTP/127.3.240.212@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/HTTP_127.3.240.212@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 10 11:40:24 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260810 11:40:24.569830 4035 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260810 11:40:24.604672 4035 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260810 11:40:25.750181 4035 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260810 11:40:25.750290 4035 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms
I20260810 11:40:25.750325 4035 mini_ranger_kms.cc:227] Using postgres at 127.3.240.212:51159
2026-08-10 11:40:26,192 [I] DB FLAVOR :POSTGRES
2026-08-10 11:40:26,193 [I] --------- Verifying Ranger DB connection ---------
2026-08-10 11:40:26,193 [I] Checking connection
2026-08-10 11:40:26,193 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-10 11:40:26,491 [I] connection success
2026-08-10 11:40:26,492 [I] --------- Verifying Ranger DB tables ---------
2026-08-10 11:40:26,492 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-10 11:40:26,492 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-10 11:40:26,801 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-10 11:40:26,801 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-10 11:40:26,801 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-10 11:40:26,801 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.3.240.212:51159/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-10 11:40:27,135 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260810 11:40:27.710238 4035 mini_ranger_kms.cc:310] Created kms service
I20260810 11:40:27.923995 4035 mini_ranger_kms.cc:326] Created kudu user
I20260810 11:40:27.979847 4035 mini_ranger_kms.cc:343] Created rangerkms user
Aug 10 11:40:27 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362024, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:28.427520 4035 mini_ranger_kms.cc:384] Added ranger policy
I20260810 11:40:28.427887 4035 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms:/tmp/dist-test-taskagqWOB/build/asan/bin/postgresql.jar:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskagqWOB/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260810 11:40:28.427938 4035 mini_ranger_kms.cc:240] Using host: 127.3.240.212
I20260810 11:40:28.456566 4035 mini_ranger_kms.cc:276] Ranger KMS PID: 22050
I20260810 11:40:28.456728 4035 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.3.240.212:42779
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-kms/ews/webapp].
Aug 10, 2026 11:40:29 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 10 11:40:30 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786362030, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:31.270224 4035 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.700s user 0.090s sys 0.392s
I20260810 11:40:31.270503 4035 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260810 11:40:31.270707 4035 mini_ranger_kms.cc:401] 127.3.240.212:42779/kms/v1/keys
Aug 10 11:40:31 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362030, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Aug 10 11:40:31 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362030, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.3.240.212@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Aug 10 11:40:34 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:40:34.259157 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43519
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43519
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:34.650264 22124 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:34.650643 22124 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:34.650826 22124 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:34.659577 22124 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:34.659690 22124 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:34.659778 22124 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:34.659888 22124 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:34.659973 22124 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:34.670773 22124 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=127.3.240.212:42779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43519
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:34.672349 22124 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:34.674108 22124 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:34.689790 22131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:34.689865 22132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:34.690667 22141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:34.692477 22124 server_base.cc:1034] running on GCE node
I20260810 11:40:34.693406 22124 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:34.695142 22124 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:34.696411 22124 hybrid_clock.cc:648] HybridClock initialized: now 1786362034696362 us; error 40 us; skew 500 ppm
I20260810 11:40:34.696813 22124 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:34 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:34.704660 22124 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:34.706447 22124 webserver.cc:460] Webserver started at http://127.3.240.254:43457/ using document root <none> and password file <none>
I20260810 11:40:34.707234 22124 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:34.707376 22124 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:34.707700 22124 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:40:34 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:34.766851 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "2b8bd05505664eb68628dcb4f7b14302"
format_stamp: "Formatted at 2026-08-10 11:40:34 on dist-test-slave-drl0"
server_key: "d87eac46cd72e37cc40de7829833fac2"
server_key_iv: "a6d882058f1f48e48d3a3134a5796b47"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:34.767881 22124 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2b8bd05505664eb68628dcb4f7b14302"
format_stamp: "Formatted at 2026-08-10 11:40:34 on dist-test-slave-drl0"
server_key: "d87eac46cd72e37cc40de7829833fac2"
server_key_iv: "a6d882058f1f48e48d3a3134a5796b47"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:34.773708 22124 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 11:40:34.793972 22152 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:34.796279 22124 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:40:34.796707 22124 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2b8bd05505664eb68628dcb4f7b14302"
format_stamp: "Formatted at 2026-08-10 11:40:34 on dist-test-slave-drl0"
server_key: "d87eac46cd72e37cc40de7829833fac2"
server_key_iv: "a6d882058f1f48e48d3a3134a5796b47"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:34.796989 22124 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:34.828835 22124 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:34.833557 22124 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:34.833957 22124 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:34.855921 22124 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43519
I20260810 11:40:34.855943 22237 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43519 every 8 connection(s)
I20260810 11:40:34.857545 22124 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:34.861750 22238 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:34.864897 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 22124
I20260810 11:40:34.865118 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:40:34.866024 4035 external_mini_cluster.cc:1428] Setting key f254866ce758c956ee27cda8b219d0e8
I20260810 11:40:34.878506 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Bootstrap starting.
I20260810 11:40:34.883957 22238 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:34.885488 22238 log.cc:824] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:34.888761 22238 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: No bootstrap required, opened a new log
Aug 10 11:40:34 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:34.899916 22238 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:34.900333 22238 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:34.900460 22238 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8bd05505664eb68628dcb4f7b14302, State: Initialized, Role: FOLLOWER
I20260810 11:40:34.901314 22238 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:34.901638 22238 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:40:34.901818 22238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:40:34.902045 22238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:34.905349 22238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:34.905984 22238 leader_election.cc:302] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b8bd05505664eb68628dcb4f7b14302; no voters:
I20260810 11:40:34.907133 22238 leader_election.cc:288] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:40:34.908452 22243 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:34.909723 22238 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:34.910835 22241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:34.868365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42282 (local address 127.3.240.254:43519)
0810 11:40:34.883204 (+ 14839us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:34.883221 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:34.883684 (+ 463us) server_negotiation.cc:407] Connection header received
0810 11:40:34.886423 (+ 2739us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:34.886449 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:34.886832 (+ 383us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:34.887297 (+ 465us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:34.888987 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:34.891112 (+ 2125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:34.893132 (+ 2020us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:34.895006 (+ 1874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:34.895356 (+ 350us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:34.900967 (+ 5611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:34.900982 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:34.901009 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:34.901060 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:34.905612 (+ 4552us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:34.906499 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:34.906508 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:34.906517 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:34.906728 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:34.907229 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:34.907234 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:34.907236 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:34.908061 (+ 825us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:34.908354 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:34.908880 (+ 526us) server_negotiation.cc:299] Negotiation successful
0810 11:40:34.909307 (+ 427us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14522,"thread_start_us":14752,"threads_started":1}
I20260810 11:40:34.918866 22243 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 LEADER]: Becoming Leader. State: Replica: 2b8bd05505664eb68628dcb4f7b14302, State: Running, Role: LEADER
I20260810 11:40:34.919804 22243 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:34.923401 22246 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b8bd05505664eb68628dcb4f7b14302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } } }
I20260810 11:40:34.923902 22246 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:34.934314 22253 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:34.937999 22247 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b8bd05505664eb68628dcb4f7b14302. Latest consensus state: current_term: 1 leader_uuid: "2b8bd05505664eb68628dcb4f7b14302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } } }
I20260810 11:40:34.939474 22247 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:34.945585 22238 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260810 11:40:34.943575 22253 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:34.983840 22253 catalog_manager.cc:1347] Generated new cluster ID: a1fc9eb24d6e4bd8bab0bd708e483b4d
I20260810 11:40:34.984045 22253 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:35.057621 22253 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:40:35.059942 22253 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:35.078485 22253 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Generated new TSK 0
I20260810 11:40:35.080801 22253 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:40:35.952145 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:36 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260810 11:40:36.419479 22295 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:36.420104 22295 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:36.420491 22295 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:36.432327 22295 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:36.432556 22295 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:36.432796 22295 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:40:36.449316 22295 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:36.451143 22295 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:36.452934 22295 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:36.471334 22310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:36.471297 22314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:36.471953 22321 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:36.472831 22295 server_base.cc:1034] running on GCE node
I20260810 11:40:36.473547 22295 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:36.474639 22295 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:36.476080 22295 hybrid_clock.cc:648] HybridClock initialized: now 1786362036475985 us; error 74 us; skew 500 ppm
I20260810 11:40:36.476567 22295 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:36 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:36.480134 22295 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:40:36.486563 22295 webserver.cc:460] Webserver started at http://127.3.240.193:43191/ using document root <none> and password file <none>
I20260810 11:40:36.487488 22295 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:36.487661 22295 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:36.488024 22295 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:40:36 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:36.517354 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "1147538babfd4cb7a66fc874603b8453"
format_stamp: "Formatted at 2026-08-10 11:40:36 on dist-test-slave-drl0"
server_key: "4158c1929d4432b917ad47acc35c779c"
server_key_iv: "9735bf32bea870eaea897e79869be8f8"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:36.518630 22295 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "1147538babfd4cb7a66fc874603b8453"
format_stamp: "Formatted at 2026-08-10 11:40:36 on dist-test-slave-drl0"
server_key: "4158c1929d4432b917ad47acc35c779c"
server_key_iv: "9735bf32bea870eaea897e79869be8f8"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:36.533164 22295 fs_manager.cc:691] Time spent creating directory manager: real 0.014s user 0.002s sys 0.010s
I20260810 11:40:36.575736 22330 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:36.578104 22295 fs_manager.cc:725] Time spent opening block manager: real 0.028s user 0.004s sys 0.015s
I20260810 11:40:36.578419 22295 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "1147538babfd4cb7a66fc874603b8453"
format_stamp: "Formatted at 2026-08-10 11:40:36 on dist-test-slave-drl0"
server_key: "4158c1929d4432b917ad47acc35c779c"
server_key_iv: "9735bf32bea870eaea897e79869be8f8"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:36.578805 22295 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:36.620101 22295 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:36.625615 22295 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:36.626020 22295 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:36.627341 22295 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:36.629187 22295 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:36.629329 22295 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260810 11:40:36.629532 22295 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:36.629647 22295 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:40:36 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:36.896620 22295 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37833
I20260810 11:40:36.896637 22500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37833 every 8 connection(s)
I20260810 11:40:36.898802 22295 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:40:36.906380 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 22295
I20260810 11:40:36.906589 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:40:36.907629 4035 external_mini_cluster.cc:1428] Setting key 6b72ebb8b76e18933d876d86e9765db6
Aug 10 11:40:36 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:36.959008 22507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:36.909318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:54653 (local address 127.3.240.254:43519)
0810 11:40:36.927952 (+ 18634us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:36.927959 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:36.927982 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:40:36.928076 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:36.928081 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:36.928223 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:36.928396 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:36.935170 (+ 6774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:36.937349 (+ 2179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:36.940207 (+ 2858us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:36.941089 (+ 882us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:36.941237 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:36.946367 (+ 5130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:36.946377 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:36.946385 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:36.946446 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:36.954190 (+ 7744us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:36.955355 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:36.955363 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:36.955370 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:36.955664 (+ 294us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:36.956334 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:36.956342 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:36.956348 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:36.956788 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:36.956947 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:36.958134 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0810 11:40:36.958440 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17589,"thread_start_us":17607,"threads_started":1}
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
I20260810 11:40:36.962424 22506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:36.908333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.193:54653)
0810 11:40:36.923133 (+ 14800us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:36.923211 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:36.926136 (+ 2925us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:36.932757 (+ 6621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:36.932773 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:36.933598 (+ 825us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:36.934671 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:36.934700 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:36.937613 (+ 2913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:36.937618 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:36.940047 (+ 2429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:36.940077 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:36.941584 (+ 1507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:36.941593 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:36.941926 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:36.942746 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:36.942783 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:36.946072 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:36.954407 (+ 8335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:36.954421 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:36.954445 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:36.955200 (+ 755us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:36.955853 (+ 653us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:36.955860 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:36.955867 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:36.956189 (+ 322us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:36.957066 (+ 877us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:36.957078 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:36.957610 (+ 532us) client_negotiation.cc:769] Sending connection context
0810 11:40:36.960091 (+ 2481us) client_negotiation.cc:240] Negotiation successful
0810 11:40:36.960747 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14361,"thread_start_us":14108,"threads_started":1}
I20260810 11:40:36.964278 22504 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:36.964676 22504 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:36.965523 22504 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
I20260810 11:40:36.968288 22177 ts_manager.cc:194] Registered new tserver with Master: 1147538babfd4cb7a66fc874603b8453 (127.3.240.193:37833)
I20260810 11:40:36.970530 22177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:54653
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:40:37.006755 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:37 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362036, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
W20260810 11:40:37.523330 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:37.523809 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:37.524025 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:37.535637 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:37.535835 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:37.536068 22511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:40:37.551388 22511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:37.553258 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:37.556025 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:37.573844 22546 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:37.575511 22544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:37.580755 22548 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:37.581811 22511 server_base.cc:1034] running on GCE node
I20260810 11:40:37.582702 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:37.584017 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:37.585235 22511 hybrid_clock.cc:648] HybridClock initialized: now 1786362037585177 us; error 45 us; skew 500 ppm
I20260810 11:40:37.585628 22511 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:37 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:37.589969 22511 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:40:37.591951 22511 webserver.cc:460] Webserver started at http://127.3.240.194:37867/ using document root <none> and password file <none>
I20260810 11:40:37.592895 22511 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:37.593032 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:37.593369 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:40:37 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:37.617416 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "67bbec121b3240df9b6c854a08635909"
format_stamp: "Formatted at 2026-08-10 11:40:37 on dist-test-slave-drl0"
server_key: "7d835fc4c9700970e082a51ed351bbfc"
server_key_iv: "25585535a480832af0455e822e93e3ec"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:37.618873 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "67bbec121b3240df9b6c854a08635909"
format_stamp: "Formatted at 2026-08-10 11:40:37 on dist-test-slave-drl0"
server_key: "7d835fc4c9700970e082a51ed351bbfc"
server_key_iv: "25585535a480832af0455e822e93e3ec"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:37.640199 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.021s user 0.006s sys 0.011s
I20260810 11:40:37.671617 22564 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:37.673940 22511 fs_manager.cc:725] Time spent opening block manager: real 0.016s user 0.011s sys 0.003s
I20260810 11:40:37.674273 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "67bbec121b3240df9b6c854a08635909"
format_stamp: "Formatted at 2026-08-10 11:40:37 on dist-test-slave-drl0"
server_key: "7d835fc4c9700970e082a51ed351bbfc"
server_key_iv: "25585535a480832af0455e822e93e3ec"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:37.674652 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:37.765053 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:37.770727 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:37.771175 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:37.772527 22511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:37.774111 22511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:37.774219 22511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:37.774380 22511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:37.774461 22511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:37.821973 22511 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:45621
I20260810 11:40:37.821991 22734 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:45621 every 8 connection(s)
I20260810 11:40:37.823534 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:40:37.829905 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 22511
I20260810 11:40:37.830116 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:40:37.830833 4035 external_mini_cluster.cc:1428] Setting key 57a975eee35a235acaa88f34f97b91d6
Aug 10 11:40:37 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362037, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:37.864367 22737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:37.827291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:48737 (local address 127.3.240.254:43519)
0810 11:40:37.840593 (+ 13302us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:37.840601 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:37.843749 (+ 3148us) server_negotiation.cc:407] Connection header received
0810 11:40:37.843824 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:37.843831 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:37.843920 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:37.844040 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:37.847110 (+ 3070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:37.848416 (+ 1306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:37.851157 (+ 2741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:37.851809 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:37.851917 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:37.856782 (+ 4865us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:37.856789 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:37.856794 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:37.856838 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:37.860490 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:37.861490 (+ 1000us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:37.861496 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:37.861500 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:37.861640 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:37.862183 (+ 543us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:37.862188 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:37.862191 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:37.862601 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:37.862732 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:37.863724 (+ 992us) server_negotiation.cc:299] Negotiation successful
0810 11:40:37.863972 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13176,"thread_start_us":10932,"threads_started":1}
I20260810 11:40:37.866945 22738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:37.827527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.194:48737)
0810 11:40:37.839390 (+ 11863us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:37.840695 (+ 1305us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:37.843326 (+ 2631us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:37.844562 (+ 1236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:37.844575 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:37.845284 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:37.846654 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:37.846705 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:37.848799 (+ 2094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:37.848805 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:37.850923 (+ 2118us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:37.850981 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:37.852295 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:37.852315 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:37.852624 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:37.853482 (+ 858us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:37.853515 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:37.856522 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:37.860692 (+ 4170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:37.860704 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:37.860724 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:37.861334 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:37.861778 (+ 444us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:37.861783 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:37.861787 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:37.862048 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:37.862831 (+ 783us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:37.862839 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:37.863295 (+ 456us) client_negotiation.cc:769] Sending connection context
0810 11:40:37.865011 (+ 1716us) client_negotiation.cc:240] Negotiation successful
0810 11:40:37.865558 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11299,"thread_start_us":11857,"threads_started":1}
I20260810 11:40:37.868723 22735 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:37.869047 22735 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:37.869910 22735 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:37.871716 22177 ts_manager.cc:194] Registered new tserver with Master: 67bbec121b3240df9b6c854a08635909 (127.3.240.194:45621)
I20260810 11:40:37.872545 22177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:48737
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:40:37.931730 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:37.974751 22504 heartbeater.cc:503] Master 127.3.240.254:43519 was elected leader, sending a full tablet report...
W20260810 11:40:38.308110 22743 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:38.308379 22743 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:38.308476 22743 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:38.316535 22743 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:38.316645 22743 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:38.316758 22743 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:40:38.328625 22743 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:38.330401 22743 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:38.332129 22743 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:38.346495 22758 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:38.346622 22757 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:38.346594 22762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:38.347707 22743 server_base.cc:1034] running on GCE node
I20260810 11:40:38.348351 22743 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:38.349587 22743 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:38.350893 22743 hybrid_clock.cc:648] HybridClock initialized: now 1786362038350811 us; error 66 us; skew 500 ppm
I20260810 11:40:38.351400 22743 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:38.354681 22743 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:40:38.356561 22743 webserver.cc:460] Webserver started at http://127.3.240.195:37559/ using document root <none> and password file <none>
I20260810 11:40:38.357239 22743 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:38.357374 22743 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:38.357708 22743 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:38.376593 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "082d07d6b973462b9aa56b67d3d505d6"
format_stamp: "Formatted at 2026-08-10 11:40:38 on dist-test-slave-drl0"
server_key: "8eb2d20912f6c56db1d1868fc1ffbc4f"
server_key_iv: "4cd328c8f38ea71ee832a3cfbbff8425"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:38.377416 22743 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "082d07d6b973462b9aa56b67d3d505d6"
format_stamp: "Formatted at 2026-08-10 11:40:38 on dist-test-slave-drl0"
server_key: "8eb2d20912f6c56db1d1868fc1ffbc4f"
server_key_iv: "4cd328c8f38ea71ee832a3cfbbff8425"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:38.383554 22743 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:40:38.402729 22774 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:38.404511 22743 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 11:40:38.404709 22743 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "082d07d6b973462b9aa56b67d3d505d6"
format_stamp: "Formatted at 2026-08-10 11:40:38 on dist-test-slave-drl0"
server_key: "8eb2d20912f6c56db1d1868fc1ffbc4f"
server_key_iv: "4cd328c8f38ea71ee832a3cfbbff8425"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:38.404973 22743 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:38.417835 22743 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:38.422500 22743 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:38.422858 22743 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:38.424293 22743 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:38.425721 22743 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:38.425804 22743 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:38.425941 22743 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:38.425994 22743 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:38.464496 22743 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41971
I20260810 11:40:38.464515 22941 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41971 every 8 connection(s)
I20260810 11:40:38.465802 22743 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:40:38.468448 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 22743
I20260810 11:40:38.468645 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:40:38.469237 4035 external_mini_cluster.cc:1428] Setting key a498f82338dcef479bfbaca5ebd59665
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362038, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:38.489745 22945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:38.470407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38605 (local address 127.3.240.254:43519)
0810 11:40:38.473361 (+ 2954us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:38.473367 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:38.473400 (+ 33us) server_negotiation.cc:407] Connection header received
0810 11:40:38.473661 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:38.473667 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:38.473762 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:38.473879 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:38.475402 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.476146 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.477703 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.478131 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.478271 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:38.482770 (+ 4499us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:38.482776 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:38.482780 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:38.482819 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:38.486287 (+ 3468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.487306 (+ 1019us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.487311 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.487313 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.487451 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.488023 (+ 572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.488029 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.488032 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.488332 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:38.488457 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:38.489232 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 11:40:38.489431 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2834,"thread_start_us":2708,"threads_started":1}
I20260810 11:40:38.490583 22944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:38.469614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.195:38605)
0810 11:40:38.470962 (+ 1348us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:38.471015 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:38.473375 (+ 2360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:38.474119 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:38.474127 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:38.474575 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:38.475229 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:38.475248 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.476309 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.476320 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:38.477536 (+ 1216us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:38.477556 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.478368 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.478377 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:38.478621 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:38.479499 (+ 878us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:38.479548 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:38.482518 (+ 2970us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:38.486458 (+ 3940us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:38.486465 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:38.486477 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:38.487152 (+ 675us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:38.487587 (+ 435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:38.487590 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:38.487593 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:38.487871 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:38.488496 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:38.488502 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:38.488806 (+ 304us) client_negotiation.cc:769] Sending connection context
0810 11:40:38.489038 (+ 232us) client_negotiation.cc:240] Negotiation successful
0810 11:40:38.489428 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":982,"thread_start_us":517,"threads_started":1}
I20260810 11:40:38.492197 22942 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:38.492504 22942 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:38.493367 22942 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:38.495013 22177 ts_manager.cc:194] Registered new tserver with Master: 082d07d6b973462b9aa56b67d3d505d6 (127.3.240.195:41971)
I20260810 11:40:38.495855 22177 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:38605
I20260810 11:40:38.498651 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:40:38.500494 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 22295
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:40:38.520862 22947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:38.501835 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41726 (local address 127.3.240.193:37833)
0810 11:40:38.502588 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:38.502595 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:38.502620 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:40:38.502740 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:38.502748 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:38.503030 (+ 282us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:38.503194 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:38.505085 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.506088 (+ 1003us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.507745 (+ 1657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.508285 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.508449 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:38.513370 (+ 4921us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:38.513381 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:38.513394 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:38.513432 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:38.517058 (+ 3626us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.517842 (+ 784us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.517850 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.517852 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.518031 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.518741 (+ 710us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.518745 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.518747 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.519182 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:38.519472 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:38.520065 (+ 593us) server_negotiation.cc:299] Negotiation successful
0810 11:40:38.520477 (+ 412us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":421,"threads_started":1}
I20260810 11:40:38.522385 22492 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:38.607685 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 22295
I20260810 11:40:38.635344 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 22511
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:40:38.659458 22949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:38.636716 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35326 (local address 127.3.240.194:45621)
0810 11:40:38.637444 (+ 728us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:38.637451 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:38.637480 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:40:38.637542 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:38.637548 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:38.637767 (+ 219us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:38.637956 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:38.639509 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.640608 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.642123 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.642673 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.642908 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:38.651709 (+ 8801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:38.651722 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:38.651738 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:38.651777 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:38.655634 (+ 3857us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.656662 (+ 1028us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.656670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.656673 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.656865 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.657369 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.657373 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.657375 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.657987 (+ 612us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:38.658311 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:38.658761 (+ 450us) server_negotiation.cc:299] Negotiation successful
0810 11:40:38.659137 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":538,"thread_start_us":388,"threads_started":1}
I20260810 11:40:38.661002 22729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:38.748009 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 22511
I20260810 11:40:38.772231 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 22743
Aug 10 11:40:38 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362034, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:40:38.792811 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:38.773594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42690 (local address 127.3.240.195:41971)
0810 11:40:38.774293 (+ 699us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:38.774301 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:38.774340 (+ 39us) server_negotiation.cc:407] Connection header received
0810 11:40:38.774413 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:38.774421 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:38.774669 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:38.774845 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:38.776383 (+ 1538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.777542 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.779073 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:38.779621 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:38.779731 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:38.785210 (+ 5479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:38.785234 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:38.785258 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:38.785313 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:38.789073 (+ 3760us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.789899 (+ 826us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.789908 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.789911 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.790051 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:38.790501 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:38.790505 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:38.790507 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:38.791284 (+ 777us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:38.791712 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:38.791968 (+ 256us) server_negotiation.cc:299] Negotiation successful
0810 11:40:38.792357 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":313,"threads_started":1}
I20260810 11:40:38.794486 22934 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:38.889531 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 22743
I20260810 11:40:38.918395 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 22124
I20260810 11:40:38.920437 22230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:39.053592 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 22124
I20260810 11:40:39.089488 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43519
--webserver_interface=127.3.240.254
--webserver_port=43457
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43519
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:39.467603 22959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:39.468400 22959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:39.468637 22959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:39.477269 22959 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:39.477401 22959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:39.477481 22959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:39.477576 22959 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:39.477623 22959 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:39.488365 22959 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43519
--ranger_config_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=43457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:39.489755 22959 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:39.491813 22959 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:39.506959 22970 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:39.507167 22967 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:39.507126 22968 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:39.508306 22959 server_base.cc:1034] running on GCE node
I20260810 11:40:39.509150 22959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:39.510600 22959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:39.511842 22959 hybrid_clock.cc:648] HybridClock initialized: now 1786362039511745 us; error 92 us; skew 500 ppm
I20260810 11:40:39.512286 22959 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:39 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:39.516079 22959 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:39.518258 22959 webserver.cc:460] Webserver started at http://127.3.240.254:43457/ using document root <none> and password file <none>
I20260810 11:40:39.519222 22959 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:39.519397 22959 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 11:40:39 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362039, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:39.567555 22959 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 11:40:39.571386 22981 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:39.573258 22959 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:40:39.573487 22959 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2b8bd05505664eb68628dcb4f7b14302"
format_stamp: "Formatted at 2026-08-10 11:40:34 on dist-test-slave-drl0"
server_key: "d87eac46cd72e37cc40de7829833fac2"
server_key_iv: "a6d882058f1f48e48d3a3134a5796b47"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:39.574487 22959 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:39.589062 22959 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:39.593472 22959 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:39.593932 22959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:39.617125 22959 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43519
I20260810 11:40:39.617225 23065 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43519 every 8 connection(s)
I20260810 11:40:39.618690 22959 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:39.624557 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 22959
I20260810 11:40:39.625717 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:37833
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:43519
--webserver_port=43191
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:39.626987 23066 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:40:39.633253 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Bootstrap starting.
I20260810 11:40:39.656473 23066 log.cc:824] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:39.667635 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:40:39.669018 23066 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Bootstrap complete.
I20260810 11:40:39.680197 23066 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:39.681299 23066 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b8bd05505664eb68628dcb4f7b14302, State: Initialized, Role: FOLLOWER
I20260810 11:40:39.681914 23066 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:39.682164 23066 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:40:39.682325 23066 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:40:39.682513 23066 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:40:39.686553 23066 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:39.687142 23066 leader_election.cc:302] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b8bd05505664eb68628dcb4f7b14302; no voters:
I20260810 11:40:39.687888 23066 leader_election.cc:288] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 11:40:39.688056 23071 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 11:40:39.689548 23071 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [term 2 LEADER]: Becoming Leader. State: Replica: 2b8bd05505664eb68628dcb4f7b14302, State: Running, Role: LEADER
I20260810 11:40:39.690239 23071 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } }
I20260810 11:40:39.690644 23066 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:39.693894 23075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b8bd05505664eb68628dcb4f7b14302. Latest consensus state: current_term: 2 leader_uuid: "2b8bd05505664eb68628dcb4f7b14302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } } }
I20260810 11:40:39.694509 23075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:39.695218 23073 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2b8bd05505664eb68628dcb4f7b14302" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b8bd05505664eb68628dcb4f7b14302" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43519 } } }
I20260810 11:40:39.695613 23073 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:39.702705 23082 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:39.710033 23082 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:39.711735 23082 catalog_manager.cc:1259] Loaded cluster ID: a1fc9eb24d6e4bd8bab0bd708e483b4d
I20260810 11:40:39.711910 23082 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:39.716578 23082 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:39.718227 23082 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Loaded TSK: 0
I20260810 11:40:39.719919 23082 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:40:39.723055 23066 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260810 11:40:40.129647 23069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:40.130084 23069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:40.130309 23069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:40.142406 23069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:40.142578 23069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:40.142768 23069 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:40:40.155447 23069 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:37833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=43191
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:40.157383 23069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:40.159355 23069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:40.189707 23122 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:40.195019 23123 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:40.204088 23126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:40.204806 23069 server_base.cc:1034] running on GCE node
I20260810 11:40:40.205713 23069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:40.207046 23069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:40.208298 23069 hybrid_clock.cc:648] HybridClock initialized: now 1786362040208191 us; error 98 us; skew 500 ppm
I20260810 11:40:40.208664 23069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:40 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:40.215523 23069 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:40:40.228720 23069 webserver.cc:460] Webserver started at http://127.3.240.193:43191/ using document root <none> and password file <none>
I20260810 11:40:40.229746 23069 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:40.229938 23069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 11:40:40 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:40.264580 23069 fs_manager.cc:709] Time spent opening directory manager: real 0.011s user 0.005s sys 0.005s
I20260810 11:40:40.289613 23138 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:40.292107 23069 fs_manager.cc:725] Time spent opening block manager: real 0.025s user 0.008s sys 0.011s
I20260810 11:40:40.292502 23069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "1147538babfd4cb7a66fc874603b8453"
format_stamp: "Formatted at 2026-08-10 11:40:36 on dist-test-slave-drl0"
server_key: "4158c1929d4432b917ad47acc35c779c"
server_key_iv: "9735bf32bea870eaea897e79869be8f8"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:40.294621 23069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:40.329487 23069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:40.336077 23069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:40.336410 23069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:40.337651 23069 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:40.339574 23069 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:40.339687 23069 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:40.339843 23069 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:40.339928 23069 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:40.411399 23069 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37833
I20260810 11:40:40.411444 23300 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37833 every 8 connection(s)
I20260810 11:40:40.413339 23069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:40:40.414710 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 23069
I20260810 11:40:40.415604 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:45621
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:43519
--webserver_port=37867
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:40 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:40.464274 23304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:40.417017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.193:52369)
0810 11:40:40.428361 (+ 11344us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:40.428425 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:40.432096 (+ 3671us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:40.434737 (+ 2641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:40.434743 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:40.435255 (+ 512us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:40.435788 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:40.435816 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:40.439190 (+ 3374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:40.439196 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:40.441328 (+ 2132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:40.441348 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:40.443341 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:40.443351 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:40.443637 (+ 286us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:40.444500 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:40.444532 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:40.449692 (+ 5160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:40.456660 (+ 6968us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:40.456674 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:40.456695 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:40.457701 (+ 1006us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:40.458338 (+ 637us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:40.458346 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:40.458352 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:40.458622 (+ 270us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:40.459988 (+ 1366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:40.459998 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:40.460765 (+ 767us) client_negotiation.cc:769] Sending connection context
0810 11:40:40.461457 (+ 692us) client_negotiation.cc:240] Negotiation successful
0810 11:40:40.462373 (+ 916us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9869,"thread_start_us":9855,"threads_started":1}
I20260810 11:40:40.466702 23305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:40.416927 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52369 (local address 127.3.240.254:43519)
0810 11:40:40.432991 (+ 16064us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:40.433005 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:40.433045 (+ 40us) server_negotiation.cc:407] Connection header received
0810 11:40:40.433370 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:40.433387 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:40.433831 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:40.434211 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:40.436002 (+ 1791us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:40.438592 (+ 2590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:40.441555 (+ 2963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:40.442630 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:40.442840 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:40.451233 (+ 8393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:40.451248 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:40.451273 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:40.451346 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:40.456431 (+ 5085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:40.457913 (+ 1482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:40.457923 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:40.457932 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:40.458160 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:40.459044 (+ 884us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:40.459055 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:40.459059 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:40.459807 (+ 748us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:40.460039 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:40.464983 (+ 4944us) server_negotiation.cc:299] Negotiation successful
0810 11:40:40.465437 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15810,"thread_start_us":15596,"threads_started":1}
I20260810 11:40:40.470484 23302 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:40.470832 23302 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:40.471807 23302 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
W20260810 11:40:40.475286 23302 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:43519 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 11:40:40.475546 23302 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:40.475826 23302 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:40.477139 23302 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:40.477392 23302 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
W20260810 11:40:40.478818 23302 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:40:40.857580 23308 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:40.857893 23308 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:40.857997 23308 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:40.866523 23308 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:40.866755 23308 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:40.867025 23308 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:40:40.885042 23308 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:45621
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=37867
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:40.887643 23308 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:40.889492 23308 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:40.907500 23333 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:40.908634 23332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:40.913257 23335 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:40.917229 23308 server_base.cc:1034] running on GCE node
I20260810 11:40:40.918282 23308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:40.919847 23308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:40.921182 23308 hybrid_clock.cc:648] HybridClock initialized: now 1786362040921082 us; error 76 us; skew 500 ppm
I20260810 11:40:40.921669 23308 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:40 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:40.925676 23308 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:40:40.927809 23308 webserver.cc:460] Webserver started at http://127.3.240.194:37867/ using document root <none> and password file <none>
I20260810 11:40:40.929113 23308 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:40.929309 23308 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 11:40:40 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:40.989009 23308 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 11:40:40.993078 23354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:40.994916 23308 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:40:40.995175 23308 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "67bbec121b3240df9b6c854a08635909"
format_stamp: "Formatted at 2026-08-10 11:40:37 on dist-test-slave-drl0"
server_key: "7d835fc4c9700970e082a51ed351bbfc"
server_key_iv: "25585535a480832af0455e822e93e3ec"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:40.996728 23308 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:41.043774 23308 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:41.050599 23308 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:41.051080 23308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:41.052536 23308 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:41.054577 23308 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:41.054729 23308 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:41.054870 23308 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:41.054993 23308 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:41.155048 23308 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:45621
I20260810 11:40:41.155109 23523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:45621 every 8 connection(s)
I20260810 11:40:41.156584 23308 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:40:41.162055 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 23308
I20260810 11:40:41.162889 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.3.240.212:42779/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:41971
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:43519
--webserver_port=37559
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38013
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362040, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:41.185303 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.161354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:53301 (local address 127.3.240.254:43519)
0810 11:40:41.163319 (+ 1965us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:41.163325 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:41.163352 (+ 27us) server_negotiation.cc:407] Connection header received
0810 11:40:41.165444 (+ 2092us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:41.165452 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:41.165556 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:41.165693 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:41.167839 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.168771 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.171477 (+ 2706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.172304 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.172455 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:41.177278 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:41.177288 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:41.177296 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:41.177351 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:41.181040 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.182157 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.182163 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.182167 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.182313 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.183007 (+ 694us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.183013 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.183017 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.183539 (+ 522us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:41.183745 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:41.184625 (+ 880us) server_negotiation.cc:299] Negotiation successful
0810 11:40:41.184897 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1823,"thread_start_us":5268,"threads_started":1}
I20260810 11:40:41.188057 23528 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.160902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.194:53301)
0810 11:40:41.162222 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:41.162304 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:41.165109 (+ 2805us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:41.165954 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:41.165963 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:41.166497 (+ 534us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:41.167513 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:41.167555 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.168979 (+ 1424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.168983 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:41.171222 (+ 2239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:41.171245 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.172823 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.172833 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:41.173143 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:41.173955 (+ 812us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:41.173985 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:41.176793 (+ 2808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:41.181232 (+ 4439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:41.181244 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:41.181261 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:41.182006 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:41.182438 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:41.182443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:41.182447 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:41.182837 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:41.183702 (+ 865us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:41.183710 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:41.184133 (+ 423us) client_negotiation.cc:769] Sending connection context
0810 11:40:41.185885 (+ 1752us) client_negotiation.cc:240] Negotiation successful
0810 11:40:41.186485 (+ 600us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":925,"thread_start_us":607,"threads_started":1}
I20260810 11:40:41.190162 23526 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:41.190503 23526 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:41.191486 23526 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:41.193859 23006 ts_manager.cc:194] Registered new tserver with Master: 67bbec121b3240df9b6c854a08635909 (127.3.240.194:45621)
I20260810 11:40:41.196388 23006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:53301
I20260810 11:40:41.479087 23302 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:41.479681 23302 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:41.481909 23006 ts_manager.cc:194] Registered new tserver with Master: 1147538babfd4cb7a66fc874603b8453 (127.3.240.193:37833)
I20260810 11:40:41.483285 23006 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:52369
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
W20260810 11:40:41.661470 23533 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:41.662086 23533 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:41.662293 23533 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:41.675777 23533 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:41.676103 23533 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:41.676470 23533 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:40:41.693454 23533 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38013
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.3.240.212:42779/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:41971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=37559
--tserver_master_addrs=127.3.240.254:43519
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:41.695238 23533 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:41.696781 23533 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:41.723819 23542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:41.727624 23541 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:41.728578 23544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:41.728833 23533 server_base.cc:1034] running on GCE node
I20260810 11:40:41.729676 23533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:41.730799 23533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:41.732086 23533 hybrid_clock.cc:648] HybridClock initialized: now 1786362041732007 us; error 63 us; skew 500 ppm
I20260810 11:40:41.732462 23533 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:41.736090 23533 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:40:41.738498 23533 webserver.cc:460] Webserver started at http://127.3.240.195:37559/ using document root <none> and password file <none>
I20260810 11:40:41.739502 23533 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:41.739699 23533 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
I20260810 11:40:41.765121 23533 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.005s sys 0.001s
I20260810 11:40:41.769958 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:41.772334 23533 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260810 11:40:41.772552 23533 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "082d07d6b973462b9aa56b67d3d505d6"
format_stamp: "Formatted at 2026-08-10 11:40:38 on dist-test-slave-drl0"
server_key: "8eb2d20912f6c56db1d1868fc1ffbc4f"
server_key_iv: "4cd328c8f38ea71ee832a3cfbbff8425"
server_key_version: "kuduclusterkey@0"
I20260810 11:40:41.774480 23533 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:41.810906 23533 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:41.816004 23533 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:41.816323 23533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:41.817588 23533 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:41.819293 23533 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:41.819408 23533 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:41.819511 23533 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:41.819585 23533 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:41.894300 23533 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:41971
I20260810 11:40:41.894413 23726 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:41971 every 8 connection(s)
I20260810 11:40:41.896365 23533 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:40:41.897924 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 23533
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for HTTP/127.3.240.212@KRBTEST.COM
Aug 10 11:40:41 dist-test-slave-drl0 krb5kdc[21465](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362041, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:41.950153 23738 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.912951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:43641 (local address 127.3.240.254:43519)
0810 11:40:41.925241 (+ 12290us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:41.925255 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:41.925274 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:40:41.925335 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:41.925340 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:41.925447 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:41.925552 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:41.928045 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.929411 (+ 1366us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.934978 (+ 5567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.935636 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.935763 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:41.940663 (+ 4900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:41.940671 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:41.940677 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:41.940740 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:41.945222 (+ 4482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.946300 (+ 1078us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.946306 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.946310 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.946469 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.947097 (+ 628us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.947102 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.947107 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.947548 (+ 441us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:41.947694 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:41.949284 (+ 1590us) server_negotiation.cc:299] Negotiation successful
0810 11:40:41.949614 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9589,"thread_start_us":8752,"threads_started":1}
I20260810 11:40:41.952284 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.899595 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42290 (local address 127.3.240.254:43519)
0810 11:40:41.916575 (+ 16980us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:41.916582 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:41.916602 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:40:41.916671 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:41.916677 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:41.916755 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:41.916867 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:41.919401 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.920618 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.925249 (+ 4631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.925874 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.925998 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:41.932226 (+ 6228us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:41.932234 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:41.932239 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:41.932291 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:41.946718 (+ 14427us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.948712 (+ 1994us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.948719 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.948723 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.948951 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:41.949867 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:41.949875 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:41.949879 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:41.950345 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:41.950683 (+ 338us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:41.951603 (+ 920us) server_negotiation.cc:299] Negotiation successful
0810 11:40:41.951879 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16846,"thread_start_us":17365,"threads_started":1}
I20260810 11:40:41.953460 23737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.912283 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43519 (local address 127.3.240.195:43641)
0810 11:40:41.916607 (+ 4324us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:41.916660 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:41.922168 (+ 5508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:41.926155 (+ 3987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:41.926170 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:41.926838 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:41.927764 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:41.927790 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.929603 (+ 1813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.929609 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:41.934779 (+ 5170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:41.934795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:41.936102 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:41.936109 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:41.936431 (+ 322us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:41.937323 (+ 892us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:41.937357 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:41.940356 (+ 2999us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:41.945451 (+ 5095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:41.945461 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:41.945481 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:41.946147 (+ 666us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:41.946635 (+ 488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:41.946642 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:41.946648 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:41.946953 (+ 305us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:41.947813 (+ 860us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:41.947821 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:41.948630 (+ 809us) client_negotiation.cc:769] Sending connection context
0810 11:40:41.950935 (+ 2305us) client_negotiation.cc:240] Negotiation successful
0810 11:40:41.951741 (+ 806us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3911,"thread_start_us":3643,"threads_started":1}
I20260810 11:40:41.955313 23727 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43519
I20260810 11:40:41.955653 23727 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:41.956480 23727 heartbeater.cc:511] Master 127.3.240.254:43519 requested a full tablet report, sending...
I20260810 11:40:41.958247 23004 ts_manager.cc:194] Registered new tserver with Master: 082d07d6b973462b9aa56b67d3d505d6 (127.3.240.195:41971)
I20260810 11:40:41.959537 23004 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:43641
I20260810 11:40:41.971788 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:40:42.022341 23736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:41.996641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42292 (local address 127.3.240.254:43519)
0810 11:40:41.996863 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:41.996869 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.005009 (+ 8140us) server_negotiation.cc:407] Connection header received
0810 11:40:42.005560 (+ 551us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.005568 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.005662 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.005743 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:42.008146 (+ 2403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.009249 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.011267 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.011953 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.012051 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.016132 (+ 4081us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:42.016138 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:42.016141 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:42.016184 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:42.018910 (+ 2726us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.019852 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.019858 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.019862 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.020013 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.020510 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.020515 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.020519 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.020913 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:42.021167 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.021746 (+ 579us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.021992 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:40:42.029986 23004 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:40:42.040882 23004 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:40:42.111258 23753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.066183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37833 (local address 127.0.0.1:41736)
0810 11:40:42.081713 (+ 15530us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:42.081781 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:42.081996 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:42.099894 (+ 17898us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:42.099903 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:42.099942 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:42.100474 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.100494 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.104504 (+ 4010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.104513 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.107634 (+ 3121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.107654 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.110159 (+ 2505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.110168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.110365 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.110433 (+ 68us) client_negotiation.cc:769] Sending connection context
0810 11:40:42.110564 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 11:40:42.110820 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15160,"thread_start_us":35581,"threads_started":1}
I20260810 11:40:42.112684 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.066888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41736 (local address 127.3.240.193:37833)
0810 11:40:42.097522 (+ 30634us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.097538 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.097617 (+ 79us) server_negotiation.cc:407] Connection header received
0810 11:40:42.097822 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.097838 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.098325 (+ 487us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.100727 (+ 2402us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:42.101064 (+ 337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.104287 (+ 3223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.107913 (+ 3626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.109902 (+ 1989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.111819 (+ 1917us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.112082 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.112155 (+ 73us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.112344 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30281,"thread_start_us":31755,"threads_started":1}
I20260810 11:40:42.117627 23750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.085867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45621 (local address 127.0.0.1:35336)
0810 11:40:42.099440 (+ 13573us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:42.099460 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:42.099636 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:42.100864 (+ 1228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:42.100870 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:42.100900 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:42.101357 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.101369 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.113134 (+ 11765us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.113142 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.114762 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.114776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.116779 (+ 2003us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.116785 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.116968 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.117004 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:40:42.117078 (+ 74us) client_negotiation.cc:240] Negotiation successful
0810 11:40:42.117224 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13426,"thread_start_us":15637,"threads_started":1}
I20260810 11:40:42.117425 23211 tablet_service.cc:1467] Processing CreateTablet for tablet b51796f8bcd34764b75cb89900258eda (DEFAULT_TABLE table=test-table [id=d940ba2a8d014b9681efe38381fefaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:42.118314 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.081627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42698 (local address 127.3.240.195:41971)
0810 11:40:42.095038 (+ 13411us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.095052 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.099272 (+ 4220us) server_negotiation.cc:407] Connection header received
0810 11:40:42.103591 (+ 4319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.103602 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.103928 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.104119 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:42.107200 (+ 3081us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.108771 (+ 1571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.115090 (+ 6319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.116539 (+ 1449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.116653 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.116906 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.117768 (+ 862us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.117943 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13170,"thread_start_us":13000,"threads_started":1}
I20260810 11:40:42.119710 23211 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51796f8bcd34764b75cb89900258eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:42.120630 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.086125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35336 (local address 127.3.240.194:45621)
0810 11:40:42.099859 (+ 13734us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.099869 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.099897 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:40:42.100017 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.100024 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.100295 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.100562 (+ 267us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:42.105373 (+ 4811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.107004 (+ 1631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.115016 (+ 8012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.116531 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.119685 (+ 3154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.120008 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.120145 (+ 137us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.120331 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13529,"thread_start_us":13347,"threads_started":1}
I20260810 11:40:42.121349 23759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.080860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41971 (local address 127.0.0.1:42698)
0810 11:40:42.099069 (+ 18209us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:42.099113 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:42.103048 (+ 3935us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:42.104537 (+ 1489us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:42.104545 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:42.104590 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:42.105118 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.105132 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.110516 (+ 5384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.110524 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.112916 (+ 2392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.112933 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.117108 (+ 4175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.117113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.117261 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.117290 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 11:40:42.117373 (+ 83us) client_negotiation.cc:240] Negotiation successful
0810 11:40:42.117505 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17988,"thread_start_us":1007,"threads_started":1}
I20260810 11:40:42.125514 23429 tablet_service.cc:1467] Processing CreateTablet for tablet b51796f8bcd34764b75cb89900258eda (DEFAULT_TABLE table=test-table [id=d940ba2a8d014b9681efe38381fefaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:42.128878 23631 tablet_service.cc:1467] Processing CreateTablet for tablet b51796f8bcd34764b75cb89900258eda (DEFAULT_TABLE table=test-table [id=d940ba2a8d014b9681efe38381fefaa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:42.130401 23631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51796f8bcd34764b75cb89900258eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:42.133448 23429 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b51796f8bcd34764b75cb89900258eda. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:42.142709 23761 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Bootstrap starting.
I20260810 11:40:42.149194 23761 tablet_bootstrap.cc:654] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:42.150903 23761 log.cc:824] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:42.161069 23761 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: No bootstrap required, opened a new log
I20260810 11:40:42.161511 23761 ts_tablet_manager.cc:1397] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Time spent bootstrapping tablet: real 0.019s user 0.011s sys 0.004s
I20260810 11:40:42.172621 23761 raft_consensus.cc:348] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.173285 23761 raft_consensus.cc:374] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:42.173466 23761 raft_consensus.cc:729] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1147538babfd4cb7a66fc874603b8453, State: Initialized, Role: FOLLOWER
I20260810 11:40:42.174304 23761 consensus_queue.cc:260] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.183086 23761 ts_tablet_manager.cc:1428] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Time spent starting tablet: real 0.021s user 0.017s sys 0.003s
I20260810 11:40:42.183595 23302 heartbeater.cc:503] Master 127.3.240.254:43519 was elected leader, sending a full tablet report...
I20260810 11:40:42.190719 23768 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Bootstrap starting.
I20260810 11:40:42.192420 23765 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Bootstrap starting.
I20260810 11:40:42.194741 23768 tablet_bootstrap.cc:654] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:42.196117 23768 log.cc:824] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:42.197710 23765 tablet_bootstrap.cc:654] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:42.199347 23765 log.cc:824] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:42.201617 23526 heartbeater.cc:503] Master 127.3.240.254:43519 was elected leader, sending a full tablet report...
I20260810 11:40:42.208118 23768 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: No bootstrap required, opened a new log
I20260810 11:40:42.208518 23768 ts_tablet_manager.cc:1397] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Time spent bootstrapping tablet: real 0.018s user 0.015s sys 0.000s
I20260810 11:40:42.212858 23765 tablet_bootstrap.cc:492] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: No bootstrap required, opened a new log
I20260810 11:40:42.213186 23765 ts_tablet_manager.cc:1397] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.013s
I20260810 11:40:42.219671 23768 raft_consensus.cc:348] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.220237 23768 raft_consensus.cc:374] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:42.220409 23768 raft_consensus.cc:729] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 082d07d6b973462b9aa56b67d3d505d6, State: Initialized, Role: FOLLOWER
I20260810 11:40:42.221313 23768 consensus_queue.cc:260] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.224619 23765 raft_consensus.cc:348] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.225113 23765 raft_consensus.cc:374] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:42.225430 23727 heartbeater.cc:503] Master 127.3.240.254:43519 was elected leader, sending a full tablet report...
I20260810 11:40:42.226255 23768 ts_tablet_manager.cc:1428] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
I20260810 11:40:42.226248 23765 raft_consensus.cc:729] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67bbec121b3240df9b6c854a08635909, State: Initialized, Role: FOLLOWER
I20260810 11:40:42.226971 23765 consensus_queue.cc:260] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.228791 23765 ts_tablet_manager.cc:1428] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Time spent starting tablet: real 0.015s user 0.007s sys 0.006s
I20260810 11:40:42.271689 23764 raft_consensus.cc:482] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:42.272185 23764 raft_consensus.cc:504] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.275107 23764 leader_election.cc:288] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 082d07d6b973462b9aa56b67d3d505d6 (127.3.240.195:41971), 67bbec121b3240df9b6c854a08635909 (127.3.240.194:45621)
I20260810 11:40:42.284289 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.275550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46301 (local address 127.3.240.195:41971)
0810 11:40:42.275854 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.275865 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.276725 (+ 860us) server_negotiation.cc:407] Connection header received
0810 11:40:42.277188 (+ 463us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.277200 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.277312 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.277401 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:42.278509 (+ 1108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.279678 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.282377 (+ 2699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.283272 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.283467 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.283572 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.283894 (+ 322us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.284023 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 11:40:42.284365 23789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.275588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:41971 (local address 127.3.240.193:46301)
0810 11:40:42.276502 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:42.276614 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:42.276896 (+ 282us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:42.277695 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:42.277703 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:42.277749 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:42.278319 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.278335 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.279833 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.279839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.282214 (+ 2375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.282229 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.283505 (+ 1276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.283511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.283695 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.283728 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:40:42.283866 (+ 138us) client_negotiation.cc:240] Negotiation successful
0810 11:40:42.283999 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":522,"threads_started":1}
I20260810 11:40:42.285429 23661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51796f8bcd34764b75cb89900258eda" candidate_uuid: "1147538babfd4cb7a66fc874603b8453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082d07d6b973462b9aa56b67d3d505d6" is_pre_election: true
I20260810 11:40:42.285967 23661 raft_consensus.cc:2454] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1147538babfd4cb7a66fc874603b8453 in term 0.
I20260810 11:40:42.286726 23146 leader_election.cc:302] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d07d6b973462b9aa56b67d3d505d6, 1147538babfd4cb7a66fc874603b8453; no voters:
I20260810 11:40:42.287025 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.277025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:38945 (local address 127.3.240.194:45621)
0810 11:40:42.277305 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.277311 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.279226 (+ 1915us) server_negotiation.cc:407] Connection header received
0810 11:40:42.279477 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.279486 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.279590 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.279833 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:42.280815 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.281956 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.284688 (+ 2732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.285992 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.286163 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.286239 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.286588 (+ 349us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.286725 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260810 11:40:42.287098 23790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.276829 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:45621 (local address 127.3.240.193:38945)
0810 11:40:42.279078 (+ 2249us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:42.279096 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:42.279294 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:42.280079 (+ 785us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:42.280084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:42.280116 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:42.280603 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.280616 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.282113 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.282119 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.284310 (+ 2191us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:42.284327 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.286215 (+ 1888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.286221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:42.286402 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.286437 (+ 35us) client_negotiation.cc:769] Sending connection context
0810 11:40:42.286534 (+ 97us) client_negotiation.cc:240] Negotiation successful
0810 11:40:42.286656 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2130,"thread_start_us":1897,"threads_started":1}
I20260810 11:40:42.287235 23764 raft_consensus.cc:2781] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:40:42.287396 23764 raft_consensus.cc:482] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:40:42.287523 23764 raft_consensus.cc:3037] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:42.289741 23764 raft_consensus.cc:504] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.290494 23764 leader_election.cc:288] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [CANDIDATE]: Term 1 election: Requested vote from peers 082d07d6b973462b9aa56b67d3d505d6 (127.3.240.195:41971), 67bbec121b3240df9b6c854a08635909 (127.3.240.194:45621)
I20260810 11:40:42.291181 23661 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51796f8bcd34764b75cb89900258eda" candidate_uuid: "1147538babfd4cb7a66fc874603b8453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "082d07d6b973462b9aa56b67d3d505d6"
I20260810 11:40:42.291419 23661 raft_consensus.cc:3037] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:42.291724 23461 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51796f8bcd34764b75cb89900258eda" candidate_uuid: "1147538babfd4cb7a66fc874603b8453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bbec121b3240df9b6c854a08635909"
I20260810 11:40:42.291810 23462 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b51796f8bcd34764b75cb89900258eda" candidate_uuid: "1147538babfd4cb7a66fc874603b8453" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "67bbec121b3240df9b6c854a08635909" is_pre_election: true
I20260810 11:40:42.292374 23461 raft_consensus.cc:3037] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:42.294412 23661 raft_consensus.cc:2454] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1147538babfd4cb7a66fc874603b8453 in term 1.
I20260810 11:40:42.294782 23146 leader_election.cc:302] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 082d07d6b973462b9aa56b67d3d505d6, 1147538babfd4cb7a66fc874603b8453; no voters:
I20260810 11:40:42.295094 23764 raft_consensus.cc:2781] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:42.295089 23461 raft_consensus.cc:2454] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1147538babfd4cb7a66fc874603b8453 in term 1.
I20260810 11:40:42.295655 23764 raft_consensus.cc:686] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 1 LEADER]: Becoming Leader. State: Replica: 1147538babfd4cb7a66fc874603b8453, State: Running, Role: LEADER
I20260810 11:40:42.296200 23764 consensus_queue.cc:237] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } }
I20260810 11:40:42.301123 23006 catalog_manager.cc:5495] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1147538babfd4cb7a66fc874603b8453 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "1147538babfd4cb7a66fc874603b8453" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1147538babfd4cb7a66fc874603b8453" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 } health_report { overall_health: UNKNOWN } } }
I20260810 11:40:42.332770 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.326881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41748 (local address 127.3.240.193:37833)
0810 11:40:42.327136 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.327140 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.327158 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:40:42.327272 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.327276 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.327364 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.327499 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:40:42.328027 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.328788 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.330397 (+ 1609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.330841 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.331138 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.331661 (+ 523us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:40:42.332188 (+ 527us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:40:42.332312 (+ 124us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.332484 (+ 172us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.332554 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 11:40:42.341822 23661 raft_consensus.cc:1261] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 1 FOLLOWER]: Refusing update from remote peer 1147538babfd4cb7a66fc874603b8453: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:42.342237 23461 raft_consensus.cc:1261] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 1 FOLLOWER]: Refusing update from remote peer 1147538babfd4cb7a66fc874603b8453: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:42.343012 23764 consensus_queue.cc:1035] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "082d07d6b973462b9aa56b67d3d505d6" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 41971 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:42.343369 23793 consensus_queue.cc:1035] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67bbec121b3240df9b6c854a08635909" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 45621 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:42.369566 23006 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:42.369863 23006 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42292:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:42.373862 23006 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2b8bd05505664eb68628dcb4f7b14302: Sending DeleteTablet for 3 replicas of tablet b51796f8bcd34764b75cb89900258eda
I20260810 11:40:42.375041 23211 tablet_service.cc:1514] Processing DeleteTablet for tablet b51796f8bcd34764b75cb89900258eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:42 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41736
I20260810 11:40:42.375180 23429 tablet_service.cc:1514] Processing DeleteTablet for tablet b51796f8bcd34764b75cb89900258eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:42 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35336
I20260810 11:40:42.375437 23631 tablet_service.cc:1514] Processing DeleteTablet for tablet b51796f8bcd34764b75cb89900258eda with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:42 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42698
I20260810 11:40:42.384328 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 23069
I20260810 11:40:42.387916 23821 tablet_replica.cc:330] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: stopping tablet replica
I20260810 11:40:42.388059 23817 tablet_replica.cc:330] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: stopping tablet replica
I20260810 11:40:42.388507 23821 raft_consensus.cc:2229] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:42.388590 23820 tablet_replica.cc:330] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: stopping tablet replica
I20260810 11:40:42.389056 23820 raft_consensus.cc:2229] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:42.389214 23821 raft_consensus.cc:2258] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:42.389570 23817 raft_consensus.cc:2229] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:40:42.390332 23817 raft_consensus.cc:2258] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:42.390866 23820 raft_consensus.cc:2258] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:42.392858 23821 ts_tablet_manager.cc:1905] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:42.393214 23817 ts_tablet_manager.cc:1905] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:42.400276 23821 ts_tablet_manager.cc:1918] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:42.400518 23821 log.cc:1196] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/b51796f8bcd34764b75cb89900258eda
I20260810 11:40:42.401167 23821 ts_tablet_manager.cc:1939] T b51796f8bcd34764b75cb89900258eda P 082d07d6b973462b9aa56b67d3d505d6: Deleting consensus metadata
I20260810 11:40:42.401381 23817 ts_tablet_manager.cc:1918] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:42.401607 23817 log.cc:1196] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/b51796f8bcd34764b75cb89900258eda
I20260810 11:40:42.402329 23817 ts_tablet_manager.cc:1939] T b51796f8bcd34764b75cb89900258eda P 1147538babfd4cb7a66fc874603b8453: Deleting consensus metadata
I20260810 11:40:42.403187 22986 catalog_manager.cc:4841] TS 082d07d6b973462b9aa56b67d3d505d6 (127.3.240.195:41971): tablet b51796f8bcd34764b75cb89900258eda (table test-table [id=d940ba2a8d014b9681efe38381fefaa5]) successfully deleted
I20260810 11:40:42.404803 22983 catalog_manager.cc:4841] TS 1147538babfd4cb7a66fc874603b8453 (127.3.240.193:37833): tablet b51796f8bcd34764b75cb89900258eda (table test-table [id=d940ba2a8d014b9681efe38381fefaa5]) successfully deleted
I20260810 11:40:42.405934 23820 ts_tablet_manager.cc:1905] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:42.407260 23751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.386116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41758 (local address 127.3.240.193:37833)
0810 11:40:42.386341 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.386350 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.386385 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:40:42.386456 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.386463 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.386598 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.386715 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:42.389851 (+ 3136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.391166 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.393683 (+ 2517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.394375 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.394514 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.398191 (+ 3677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:42.398206 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:42.398225 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:42.398271 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:42.403365 (+ 5094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.404694 (+ 1329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.404704 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.404711 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.404888 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.405421 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.405427 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.405431 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.405878 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:42.406240 (+ 362us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.406594 (+ 354us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.406844 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 11:40:42.408061 23296 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:42.412765 23820 ts_tablet_manager.cc:1918] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:42.412925 23820 log.cc:1196] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/b51796f8bcd34764b75cb89900258eda
I20260810 11:40:42.413406 23820 ts_tablet_manager.cc:1939] T b51796f8bcd34764b75cb89900258eda P 67bbec121b3240df9b6c854a08635909: Deleting consensus metadata
I20260810 11:40:42.414891 22986 catalog_manager.cc:4841] TS 67bbec121b3240df9b6c854a08635909 (127.3.240.194:45621): tablet b51796f8bcd34764b75cb89900258eda (table test-table [id=d940ba2a8d014b9681efe38381fefaa5]) successfully deleted
I20260810 11:40:42.515679 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 23069
I20260810 11:40:42.548626 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 23308
I20260810 11:40:42.563759 23760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.549416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35344 (local address 127.3.240.194:45621)
0810 11:40:42.549607 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.549612 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.549632 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:40:42.549949 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.549954 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.550030 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.550124 (+ 94us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:42.551637 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.552365 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.553664 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.554079 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.554186 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.557681 (+ 3495us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:42.557703 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:42.557727 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:42.557786 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:42.561058 (+ 3272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.561787 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.561795 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.561798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.561956 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.562323 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.562327 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.562329 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.562725 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:42.563037 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.563316 (+ 279us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.563483 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:40:42.564503 23517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:42.656659 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 23308
I20260810 11:40:42.684675 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 23533
I20260810 11:40:42.700358 23758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:42.685557 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42706 (local address 127.3.240.195:41971)
0810 11:40:42.685960 (+ 403us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:42.685968 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:42.685999 (+ 31us) server_negotiation.cc:407] Connection header received
0810 11:40:42.686239 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:42.686248 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:42.686336 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:42.686481 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:42.688153 (+ 1672us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.689000 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.690579 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:42.691197 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:42.691348 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:42.694370 (+ 3022us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:42.694384 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:42.694403 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:42.694436 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:42.697319 (+ 2883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.698050 (+ 731us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.698057 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.698059 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.698186 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:42.698592 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:42.698597 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:42.698601 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:42.699051 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:42.699387 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:42.699730 (+ 343us) server_negotiation.cc:299] Negotiation successful
0810 11:40:42.699973 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260810 11:40:42.701131 23721 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:42.792943 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 23533
I20260810 11:40:42.819530 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 22959
I20260810 11:40:42.821300 23057 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:42.944633 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 22959
I20260810 11:40:42.994721 4035 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260810 11:40:43.352176 4035 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260810 11:40:43.352442 4035 mini_ranger.cc:66] Stopping Ranger...
I20260810 11:40:43.699738 4035 mini_ranger.cc:68] Stopped Ranger
2026-08-10 11:40:43.700 UTC [21513] LOG: received smart shutdown request
2026-08-10 11:40:43.703 UTC [21513] LOG: background worker "logical replication launcher" (PID 21526) exited with exit code 1
2026-08-10 11:40:43.704 UTC [21520] LOG: shutting down
2026-08-10 11:40:43.904 UTC [21513] LOG: database system is shut down
2026-08-10T11:40:43Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (58616 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:40:44 dist-test-slave-drl0 krb5kdc[23839](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:40:44 dist-test-slave-drl0 krb5kdc[23839](info): set up 2 sockets
Aug 10 11:40:44 dist-test-slave-drl0 krb5kdc[23839](info): commencing operation
krb5kdc: starting...
W20260810 11:40:45.425364 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.001s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:40:45 dist-test-slave-drl0 krb5kdc[23839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:40:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:40:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:40:45.730712 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46607
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:46.113219 23854 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:46.113536 23854 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:46.113705 23854 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:46.122236 23854 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:46.122380 23854 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:46.122473 23854 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:46.122596 23854 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:46.122663 23854 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:46.133652 23854 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:46.135164 23854 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:46.136912 23854 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:46.150198 23867 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:46.150341 23865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:46.150228 23864 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:46.151064 23854 server_base.cc:1034] running on GCE node
I20260810 11:40:46.151877 23854 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:46.153492 23854 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:46.154824 23854 hybrid_clock.cc:648] HybridClock initialized: now 1786362046154762 us; error 47 us; skew 500 ppm
I20260810 11:40:46.155419 23854 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:46.162077 23854 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:46.163915 23854 webserver.cc:460] Webserver started at http://127.3.240.254:46185/ using document root <none> and password file <none>
I20260810 11:40:46.164685 23854 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:46.164844 23854 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:46.165167 23854 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:46.168323 23854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "b43b8992772b4b07ae46fd84cbc93e64"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "dbe3bd600f0ae838d6d953c6a6a6fe3b"
server_key_iv: "a997fafb6e33f47eb26103d2de10ae6f"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.169178 23854 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "b43b8992772b4b07ae46fd84cbc93e64"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "dbe3bd600f0ae838d6d953c6a6a6fe3b"
server_key_iv: "a997fafb6e33f47eb26103d2de10ae6f"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.175230 23854 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 11:40:46.179977 23878 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:46.181734 23854 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:40:46.181977 23854 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "b43b8992772b4b07ae46fd84cbc93e64"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "dbe3bd600f0ae838d6d953c6a6a6fe3b"
server_key_iv: "a997fafb6e33f47eb26103d2de10ae6f"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.182255 23854 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:46.200572 23854 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:46.206863 23854 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:46.207235 23854 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:46.229321 23854 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46607
I20260810 11:40:46.229347 23959 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46607 every 8 connection(s)
I20260810 11:40:46.230904 23854 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:46.232846 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 23854
I20260810 11:40:46.233094 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:40:46.233767 4035 external_mini_cluster.cc:1428] Setting key f1c9974a2520c212fcf379ec8c8cd411
I20260810 11:40:46.235826 23960 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:46.243386 23960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
I20260810 11:40:46.272388 23968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.247444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41888 (local address 127.3.240.254:46607)
0810 11:40:46.248680 (+ 1236us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:46.248696 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:46.249497 (+ 801us) server_negotiation.cc:407] Connection header received
0810 11:40:46.251836 (+ 2339us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:46.251846 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:46.252161 (+ 315us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:46.252376 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:46.253571 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.255629 (+ 2058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.257100 (+ 1471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.258116 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.258321 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.263218 (+ 4897us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:46.263226 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:46.263247 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:46.263280 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:46.266805 (+ 3525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.267713 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.267721 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.267723 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.267948 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.268415 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.268419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.268421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.269280 (+ 859us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:46.269452 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:46.270383 (+ 931us) server_negotiation.cc:299] Negotiation successful
0810 11:40:46.271073 (+ 690us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":200,"server-negotiator.queue_time_us":967,"thread_start_us":713,"threads_started":1}
I20260810 11:40:46.272449 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.247444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46607 (local address 127.0.0.1:41888)
0810 11:40:46.249285 (+ 1841us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.249350 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:46.251414 (+ 2064us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:46.252439 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:46.252444 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:46.253022 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:46.253363 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.253381 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.255831 (+ 2450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.255836 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.256948 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.256961 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.258368 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.258374 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.258508 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.259446 (+ 938us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:46.259468 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:46.262693 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:46.267059 (+ 4366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.267071 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.267080 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.267521 (+ 441us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.268092 (+ 571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.268096 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.268099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.268272 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.269450 (+ 1178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:46.269457 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:46.269765 (+ 308us) client_negotiation.cc:769] Sending connection context
0810 11:40:46.270078 (+ 313us) client_negotiation.cc:240] Negotiation successful
0810 11:40:46.270786 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1573,"spinlock_wait_cycles":95104,"thread_start_us":1337,"threads_started":1}
I20260810 11:40:46.276497 23960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:40:46.277892 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.277362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33340)
0810 11:40:46.277570 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.277712 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 11:40:46.278324 23881 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
I20260810 11:40:46.280768 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.280161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33342)
0810 11:40:46.280400 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.280534 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
W20260810 11:40:46.281248 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:32957: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260810 11:40:46.307452 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:32957
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:46.331073 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } attempt: 1
I20260810 11:40:46.332929 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.332204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33346)
0810 11:40:46.332526 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.332746 (+ 220us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260810 11:40:46.334438 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.333968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33362)
0810 11:40:46.334168 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.334274 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
W20260810 11:40:46.334764 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:32957: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
I20260810 11:40:46.403105 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } attempt: 2
I20260810 11:40:46.404484 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.403915 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33364)
0810 11:40:46.404224 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.404323 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260810 11:40:46.405584 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.405210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33366)
0810 11:40:46.405387 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.405465 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 11:40:46.405794 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:32957: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
I20260810 11:40:46.497092 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } attempt: 3
I20260810 11:40:46.498272 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.497838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33368)
0810 11:40:46.498046 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.498141 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 11:40:46.499573 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.499065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33372)
0810 11:40:46.499280 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.499435 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
W20260810 11:40:46.499765 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:32957: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
I20260810 11:40:46.643087 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } attempt: 4
I20260810 11:40:46.644410 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.643763 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33382)
0810 11:40:46.644119 (+ 356us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.644225 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260810 11:40:46.645596 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.645141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33398)
0810 11:40:46.645307 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.645428 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 11:40:46.645835 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:32957: Network error: Client connection negotiation failed: client connection to 127.3.240.253:32957: connect: Connection refused (error 111)
W20260810 11:40:46.686770 23974 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:46.687327 23974 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:46.687536 23974 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:46.696444 23974 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:46.696552 23974 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:46.696681 23974 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:46.696741 23974 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:46.696796 23974 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:46.707647 23974 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:32957
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:46.709625 23974 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:46.711059 23974 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:46.723901 23986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:46.724100 23980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:46.724443 23982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:46.725317 23974 server_base.cc:1034] running on GCE node
I20260810 11:40:46.726078 23974 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:46.727739 23974 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:46.729017 23974 hybrid_clock.cc:648] HybridClock initialized: now 1786362046728966 us; error 60 us; skew 500 ppm
I20260810 11:40:46.729393 23974 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:46.733194 23974 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260810 11:40:46.734874 23974 webserver.cc:460] Webserver started at http://127.3.240.253:35627/ using document root <none> and password file <none>
I20260810 11:40:46.735627 23974 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:46.735734 23974 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:46.736063 23974 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:46.739293 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data/instance:
uuid: "6f4e8e75cc79473897826bb7c0b8f5ad"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "af4d774142717c8e0324e6163fbf588b"
server_key_iv: "19ea017b8574666ce323580cbc768d81"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.740123 23974 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal/instance:
uuid: "6f4e8e75cc79473897826bb7c0b8f5ad"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "af4d774142717c8e0324e6163fbf588b"
server_key_iv: "19ea017b8574666ce323580cbc768d81"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.746120 23974 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:40:46.749998 23998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:46.751647 23974 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:40:46.751834 23974 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "6f4e8e75cc79473897826bb7c0b8f5ad"
format_stamp: "Formatted at 2026-08-10 11:40:46 on dist-test-slave-drl0"
server_key: "af4d774142717c8e0324e6163fbf588b"
server_key_iv: "19ea017b8574666ce323580cbc768d81"
server_key_version: "encryptionkey@0"
I20260810 11:40:46.752034 23974 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:46.770313 23974 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:46.775559 23974 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:46.775851 23974 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:46.797276 23974 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:32957
I20260810 11:40:46.797319 24076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:32957 every 8 connection(s)
I20260810 11:40:46.798744 23974 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
I20260810 11:40:46.800479 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 23974
I20260810 11:40:46.800707 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-1/wal/instance
I20260810 11:40:46.801380 4035 external_mini_cluster.cc:1428] Setting key 85675d6b685b56a4290ecc3c159572a1
I20260810 11:40:46.803079 24078 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:46.811748 24078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
I20260810 11:40:46.834825 24085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.814902 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41904 (local address 127.3.240.254:46607)
0810 11:40:46.815443 (+ 541us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:46.815448 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:46.816578 (+ 1130us) server_negotiation.cc:407] Connection header received
0810 11:40:46.818695 (+ 2117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:46.818702 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:46.818810 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:46.818957 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:46.820654 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.821418 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.822983 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.823394 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.823530 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.827171 (+ 3641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:46.827176 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:46.827180 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:46.827223 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:46.830446 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.831710 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.831716 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.831718 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.831910 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.832553 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.832558 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.832560 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.832981 (+ 421us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:46.833198 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:46.834029 (+ 831us) server_negotiation.cc:299] Negotiation successful
0810 11:40:46.834333 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":440,"thread_start_us":279,"threads_started":1}
I20260810 11:40:46.835655 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.815520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46607 (local address 127.0.0.1:41904)
0810 11:40:46.816387 (+ 867us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.816435 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:46.818401 (+ 1966us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:46.819210 (+ 809us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:46.819219 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:46.819906 (+ 687us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:46.820470 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.820486 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.821587 (+ 1101us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.821591 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.822797 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.822814 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.823629 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.823637 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.823862 (+ 225us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.824707 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:46.824731 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:46.826903 (+ 2172us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:46.830739 (+ 3836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.830751 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.830770 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.831531 (+ 761us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.832117 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.832122 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.832125 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.832360 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.833163 (+ 803us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:46.833169 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:46.833527 (+ 358us) client_negotiation.cc:769] Sending connection context
0810 11:40:46.833783 (+ 256us) client_negotiation.cc:240] Negotiation successful
0810 11:40:46.834379 (+ 596us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":607,"thread_start_us":371,"threads_started":1}
I20260810 11:40:46.838053 24078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:40:46.858319 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.839311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33400 (local address 127.3.240.253:32957)
0810 11:40:46.840122 (+ 811us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:46.840128 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:46.840154 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:40:46.840197 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:46.840203 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:46.840393 (+ 190us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:46.840514 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:46.841592 (+ 1078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.842707 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.844491 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.845526 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.845715 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.850050 (+ 4335us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:46.850064 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:46.850079 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:46.850110 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:46.854200 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.855505 (+ 1305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.855512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.855515 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.855658 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.856328 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.856334 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.856336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.856913 (+ 577us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:46.857148 (+ 235us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:46.857707 (+ 559us) server_negotiation.cc:299] Negotiation successful
0810 11:40:46.857992 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":504,"threads_started":1}
I20260810 11:40:46.858418 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.839171 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33400)
0810 11:40:46.839434 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.839458 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:46.839649 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:46.840561 (+ 912us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:46.840573 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:46.841142 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:46.841401 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.841415 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.842855 (+ 1440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.842862 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.844333 (+ 1471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.844350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.845796 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.845812 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.846114 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.846853 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:46.846885 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:46.849714 (+ 2829us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:46.854347 (+ 4633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.854354 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.854362 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.855265 (+ 903us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.855835 (+ 570us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.855843 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.855858 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.856055 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.857109 (+ 1054us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:46.857116 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:46.857320 (+ 204us) client_negotiation.cc:769] Sending connection context
0810 11:40:46.857487 (+ 167us) client_negotiation.cc:240] Negotiation successful
0810 11:40:46.857817 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 11:40:46.861334 24078 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:40:46.862509 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.862032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47200)
0810 11:40:46.862226 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.862348 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
W20260810 11:40:46.862886 24001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:46.864668 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.864298 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47208)
0810 11:40:46.864483 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.864559 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
W20260810 11:40:46.865033 24078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
I20260810 11:40:46.875103 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:34237
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39875
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:46.914438 24078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 1
I20260810 11:40:46.916143 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.915574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47212)
0810 11:40:46.915841 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.915964 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 11:40:46.917984 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.917374 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47228)
0810 11:40:46.917598 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.917830 (+ 232us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 11:40:46.918237 24078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:46.945142 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } attempt: 5
Aug 10 11:40:46 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:40:46.960283 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.945963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33404 (local address 127.3.240.253:32957)
0810 11:40:46.946150 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:46.946157 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:46.946230 (+ 73us) server_negotiation.cc:407] Connection header received
0810 11:40:46.946435 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:46.946441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:46.946528 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:46.946633 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:46.947476 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.948174 (+ 698us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.949329 (+ 1155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.949773 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.949900 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.953439 (+ 3539us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:46.953444 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:46.953448 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:46.953477 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:46.956763 (+ 3286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.957680 (+ 917us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.957685 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.957688 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.957850 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:46.958340 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:46.958344 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:46.958346 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:46.958737 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:46.958884 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:46.959659 (+ 775us) server_negotiation.cc:299] Negotiation successful
0810 11:40:46.959890 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 11:40:46.960379 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.945880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33404)
0810 11:40:46.946148 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.946167 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:46.946309 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:46.946661 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:46.946666 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:46.947144 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:46.947329 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.947339 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.948307 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.948315 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.949188 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:46.949199 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:46.949924 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:46.949930 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:46.950100 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:46.950806 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:46.950825 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:46.953239 (+ 2414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:46.956983 (+ 3744us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.956991 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.956999 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.957516 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.957998 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:46.958003 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:46.958007 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:46.958183 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:46.958975 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:46.958984 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:46.959255 (+ 271us) client_negotiation.cc:769] Sending connection context
0810 11:40:46.959520 (+ 265us) client_negotiation.cc:240] Negotiation successful
0810 11:40:46.959968 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148}
I20260810 11:40:46.961795 23960 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:40:46.963169 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.962496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47240)
0810 11:40:46.962736 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.962857 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 11:40:46.964388 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.963911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47242)
0810 11:40:46.964090 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.964208 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 11:40:46.964705 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:46.986492 24078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 2
I20260810 11:40:46.987850 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.987249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47252)
0810 11:40:46.987482 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.987599 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260810 11:40:46.989025 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:46.988661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47254)
0810 11:40:46.988804 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:46.988888 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
W20260810 11:40:46.989378 24078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:47.016026 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 1
I20260810 11:40:47.017793 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.016998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47264)
0810 11:40:47.017402 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.017545 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":189}
I20260810 11:40:47.019156 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.018746 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47268)
0810 11:40:47.018907 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.019038 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260810 11:40:47.019445 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:47.080679 24078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 3
I20260810 11:40:47.082114 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.081536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47280)
0810 11:40:47.081791 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.081912 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260810 11:40:47.083791 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.083191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47282)
0810 11:40:47.083514 (+ 323us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.083648 (+ 134us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":212}
W20260810 11:40:47.084085 24078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:47.087687 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 2
I20260810 11:40:47.088982 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.088348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47294)
0810 11:40:47.088599 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.088734 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 11:40:47.090227 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.089755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47296)
0810 11:40:47.089973 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.090076 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
W20260810 11:40:47.090473 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:47.196717 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 3
I20260810 11:40:47.198362 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.197695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47302)
0810 11:40:47.197994 (+ 299us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.198140 (+ 146us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 11:40:47.199683 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.199287 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47304)
0810 11:40:47.199472 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.199559 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 11:40:47.199937 23960 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
I20260810 11:40:47.227344 24078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 4
I20260810 11:40:47.228679 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.228086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47316)
0810 11:40:47.228346 (+ 260us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.228483 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260810 11:40:47.229841 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.229451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47332)
0810 11:40:47.229615 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.229700 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 11:40:47.230120 24078 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:34237: Network error: Client connection negotiation failed: client connection to 127.3.240.252:34237: connect: Connection refused (error 111)
W20260810 11:40:47.253369 24091 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:47.253651 24091 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:47.253835 24091 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:47.261808 24091 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:47.261922 24091 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:47.261970 24091 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:47.262061 24091 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:47.262137 24091 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:47.272622 24091 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39875
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46607,127.3.240.253:32957,127.3.240.252:34237
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:34237
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:47.274004 24091 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:47.275611 24091 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:47.288671 24100 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:47.288965 24101 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:47.289127 24103 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:47.290006 24091 server_base.cc:1034] running on GCE node
I20260810 11:40:47.290839 24091 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:47.292418 24091 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:47.293785 24091 hybrid_clock.cc:648] HybridClock initialized: now 1786362047293797 us; error 95 us; skew 500 ppm
I20260810 11:40:47.294155 24091 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362047, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:47.302608 24091 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260810 11:40:47.304409 24091 webserver.cc:460] Webserver started at http://127.3.240.252:40467/ using document root <none> and password file <none>
I20260810 11:40:47.305083 24091 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:47.305186 24091 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:47.305449 24091 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:47.308646 24091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data/instance:
uuid: "44234bf7f3af4bc5a9b991cce73de243"
format_stamp: "Formatted at 2026-08-10 11:40:47 on dist-test-slave-drl0"
server_key: "940326eb1273385282eb55e509843352"
server_key_iv: "ac62dba76cc5994f2415b95622fbba67"
server_key_version: "encryptionkey@0"
I20260810 11:40:47.309439 24091 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal/instance:
uuid: "44234bf7f3af4bc5a9b991cce73de243"
format_stamp: "Formatted at 2026-08-10 11:40:47 on dist-test-slave-drl0"
server_key: "940326eb1273385282eb55e509843352"
server_key_iv: "ac62dba76cc5994f2415b95622fbba67"
server_key_version: "encryptionkey@0"
I20260810 11:40:47.314669 24091 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:40:47.318681 24114 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:47.320329 24091 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:40:47.320500 24091 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "44234bf7f3af4bc5a9b991cce73de243"
format_stamp: "Formatted at 2026-08-10 11:40:47 on dist-test-slave-drl0"
server_key: "940326eb1273385282eb55e509843352"
server_key_iv: "ac62dba76cc5994f2415b95622fbba67"
server_key_version: "encryptionkey@0"
I20260810 11:40:47.320672 24091 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:47.344133 24091 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:47.348752 24091 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:47.349009 24091 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:47.370648 24091 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:34237
I20260810 11:40:47.370680 24195 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:34237 every 8 connection(s)
I20260810 11:40:47.372044 24091 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
I20260810 11:40:47.376677 24197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:47.377179 23960 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 4
I20260810 11:40:47.378535 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 24091
I20260810 11:40:47.378783 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786361809851056-4035-0/minicluster-data/master-2/wal/instance
I20260810 11:40:47.379566 4035 external_mini_cluster.cc:1428] Setting key be290cc138591278a8c17fcf23ae1978
I20260810 11:40:47.388803 24197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362047, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:47.403496 23969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.377942 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47336)
0810 11:40:47.378314 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.378335 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:47.378455 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:47.384198 (+ 5743us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:47.384203 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:47.384779 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:47.385001 (+ 222us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.385013 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.387469 (+ 2456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.387474 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.388901 (+ 1427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.388918 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.390246 (+ 1328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.390252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.390452 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.391259 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:47.391285 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:47.394652 (+ 3367us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:47.399589 (+ 4937us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.399595 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.399599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.400170 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.401018 (+ 848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.401024 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.401030 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.401211 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.402148 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:47.402156 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:47.402363 (+ 207us) client_negotiation.cc:769] Sending connection context
0810 11:40:47.402566 (+ 203us) client_negotiation.cc:240] Negotiation successful
0810 11:40:47.402900 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":236}
I20260810 11:40:47.404369 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.380747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.3.240.252:34237)
0810 11:40:47.382423 (+ 1676us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.382428 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.382449 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:40:47.383183 (+ 734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.383206 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.383637 (+ 431us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.384015 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.385673 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.387213 (+ 1540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.389288 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.390048 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.390244 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.396018 (+ 5774us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.396025 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.396030 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.396065 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.400048 (+ 3983us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.400752 (+ 704us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.400760 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.400766 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.400915 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.401388 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.401394 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.401398 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.401948 (+ 550us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.402285 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.402681 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.403160 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":1564,"spinlock_wait_cycles":15232,"thread_start_us":1320,"threads_started":1}
I20260810 11:40:47.404703 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.379580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.3.240.252:34237)
0810 11:40:47.380558 (+ 978us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.380580 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.380635 (+ 55us) server_negotiation.cc:407] Connection header received
0810 11:40:47.383171 (+ 2536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.383205 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.383641 (+ 436us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.384017 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.385342 (+ 1325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.387210 (+ 1868us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.389057 (+ 1847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.390059 (+ 1002us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.390857 (+ 798us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.395022 (+ 4165us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.395033 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.395047 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.395101 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.399431 (+ 4330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.400302 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.400312 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.400321 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.400862 (+ 541us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.401363 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.401370 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.401374 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.401958 (+ 584us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.403780 (+ 1822us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.403949 (+ 169us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.404155 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1115,"server-negotiator.queue_time_us":699,"spinlock_wait_cycles":65920,"thread_start_us":438,"threads_started":1}
I20260810 11:40:47.411365 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.391878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41906 (local address 127.3.240.254:46607)
0810 11:40:47.392460 (+ 582us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.392466 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.393142 (+ 676us) server_negotiation.cc:407] Connection header received
0810 11:40:47.393490 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.393497 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.393581 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.393699 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.395197 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.396395 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.399058 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.399777 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.399926 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.404374 (+ 4448us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.404382 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.404388 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.404429 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.408132 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.409115 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.409123 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.409129 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.409284 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.409687 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.409693 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.409696 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.410027 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.410192 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.410662 (+ 470us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.410908 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":426,"thread_start_us":1597,"threads_started":1}
I20260810 11:40:47.411439 23960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: Bootstrap starting.
I20260810 11:40:47.412042 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.391870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46607 (local address 127.0.0.1:41906)
0810 11:40:47.392981 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.393052 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:47.393289 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:47.393780 (+ 491us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:47.393788 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:47.394577 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:47.395008 (+ 431us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.395020 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.396691 (+ 1671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.396701 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.398740 (+ 2039us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.398756 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.400006 (+ 1250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.400014 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.400215 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.400964 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:47.401000 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:47.404142 (+ 3142us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:47.408358 (+ 4216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.408380 (+ 22us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.408394 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.408950 (+ 556us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.409411 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.409417 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.409421 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.409573 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.410161 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:47.410168 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:47.410362 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 11:40:47.410567 (+ 205us) client_negotiation.cc:240] Negotiation successful
0810 11:40:47.410874 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":759,"threads_started":1}
I20260810 11:40:47.413810 24197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:40:47.415858 23960 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:47.417191 23960 log.cc:824] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:47.421198 23960 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: No bootstrap required, opened a new log
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362047, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:47.428920 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.414704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:32957 (local address 127.0.0.1:33408)
0810 11:40:47.414994 (+ 290us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.415011 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:47.415098 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:47.415431 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:47.415436 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:47.415890 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:47.416066 (+ 176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.416077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.417415 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.417423 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.418277 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.418288 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.419061 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.419066 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.419196 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.419935 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:47.419963 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:47.422511 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:47.426360 (+ 3849us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.426365 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.426369 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.426863 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.427286 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.427291 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.427295 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.427453 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.427964 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:47.427969 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:47.428119 (+ 150us) client_negotiation.cc:769] Sending connection context
0810 11:40:47.428259 (+ 140us) client_negotiation.cc:240] Negotiation successful
0810 11:40:47.428546 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159}
I20260810 11:40:47.429311 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.414785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.3.240.253:32957)
0810 11:40:47.415021 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.415027 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.415051 (+ 24us) server_negotiation.cc:407] Connection header received
0810 11:40:47.415218 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.415223 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.415299 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.415403 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.416228 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.417247 (+ 1019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.418423 (+ 1176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.418848 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.419015 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.422694 (+ 3679us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.422701 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.422706 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.422767 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.426206 (+ 3439us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.427015 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.427023 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.427027 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.427169 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.427572 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.427577 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.427580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.427846 (+ 266us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.428011 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.428403 (+ 392us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.428621 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 11:40:47.430299 24197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:40:47.431878 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.411473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41914 (local address 127.3.240.254:46607)
0810 11:40:47.411669 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.411677 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.411877 (+ 200us) server_negotiation.cc:407] Connection header received
0810 11:40:47.412065 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.412073 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.412201 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.412313 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.414095 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.415218 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.416830 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.417261 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.418488 (+ 1227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.424252 (+ 5764us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.424259 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.424264 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.424301 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.428378 (+ 4077us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.429277 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.429282 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.429285 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.429428 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.430034 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.430040 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.430043 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.430456 (+ 413us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.430748 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.431087 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.431327 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73,"thread_start_us":1084,"threads_started":1}
I20260810 11:40:47.433118 23960 raft_consensus.cc:348] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.433508 23960 raft_consensus.cc:374] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:47.433627 23960 raft_consensus.cc:729] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b43b8992772b4b07ae46fd84cbc93e64, State: Initialized, Role: FOLLOWER
I20260810 11:40:47.434263 23960 consensus_queue.cc:260] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.435549 24208 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.436053 24208 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.437037 23960 sys_catalog.cc:564] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: configured and running, proceeding with master startup.
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362047, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362045, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:40:47.447521 24203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.431208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47350)
0810 11:40:47.431588 (+ 380us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.431603 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:47.431723 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:47.432163 (+ 440us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:47.432168 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:47.432642 (+ 474us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:47.432857 (+ 215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.432870 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.434103 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.434108 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.435484 (+ 1376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.435501 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.436610 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.436614 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.436725 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.437290 (+ 565us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:47.437313 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:47.439891 (+ 2578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:47.443895 (+ 4004us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.443901 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.443906 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.444655 (+ 749us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.445212 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.445218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.445222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.445401 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.446345 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:47.446351 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:47.446545 (+ 194us) client_negotiation.cc:769] Sending connection context
0810 11:40:47.446700 (+ 155us) client_negotiation.cc:240] Negotiation successful
0810 11:40:47.447085 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":303}
I20260810 11:40:47.448028 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.431115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47350 (local address 127.3.240.252:34237)
0810 11:40:47.431439 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.431446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.431670 (+ 224us) server_negotiation.cc:407] Connection header received
0810 11:40:47.431848 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.431854 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.431914 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.432016 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.433015 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.433958 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.435658 (+ 1700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.436319 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.436454 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.440053 (+ 3599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.440060 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.440064 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.440101 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.443743 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.444852 (+ 1109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.444861 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.444867 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.445062 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.445553 (+ 491us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.445563 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.445569 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.446174 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.446397 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.446880 (+ 483us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.447351 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":220}
I20260810 11:40:47.453490 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.433870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33424 (local address 127.3.240.253:32957)
0810 11:40:47.434021 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.434029 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.434273 (+ 244us) server_negotiation.cc:407] Connection header received
0810 11:40:47.434423 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.434428 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.434511 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.434733 (+ 222us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.436504 (+ 1771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.437555 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.439481 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.440318 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.440476 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.445744 (+ 5268us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.445749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.445752 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.445782 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.449173 (+ 3391us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.450379 (+ 1206us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.450386 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.450389 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.450568 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.451089 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.451093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.451096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.451462 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.452790 (+ 1328us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.452915 (+ 125us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.453142 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260810 11:40:47.454103 24197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: Bootstrap starting.
I20260810 11:40:47.458947 24197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:47.459795 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:40:47.460574 24197 log.cc:824] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: Log is configured to *not* fsync() on all Append() calls
W20260810 11:40:47.461452 23891 tablet.cc:2369] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:40:47.470161 24197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: No bootstrap required, opened a new log
I20260810 11:40:47.457595 23854 master_runner.cc:186] Error getting master registration for 127.3.240.253:32957: OK, instance_id { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" instance_seqno: 1786362046791046 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:40:47.471983 23854 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260810 11:40:47.476882 24221 catalog_manager.cc:1558] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:40:47.477018 24221 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:40:47.481634 24197 raft_consensus.cc:348] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.482407 24197 raft_consensus.cc:374] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:47.482553 24197 raft_consensus.cc:729] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44234bf7f3af4bc5a9b991cce73de243, State: Initialized, Role: FOLLOWER
I20260810 11:40:47.483426 24197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.505023 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.507057 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.506614 24197 sys_catalog.cc:564] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:47.520881 24202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.483422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41926 (local address 127.3.240.254:46607)
0810 11:40:47.483800 (+ 378us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.483810 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.496212 (+ 12402us) server_negotiation.cc:407] Connection header received
0810 11:40:47.496741 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.496749 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.496846 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.496928 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.499208 (+ 2280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.500245 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.502368 (+ 2123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.503027 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.503121 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.509786 (+ 6665us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.509794 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.509801 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.509883 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.514077 (+ 4194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.516975 (+ 2898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.516981 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.516985 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.517160 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.517674 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.517678 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.517680 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.517984 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.518252 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.519299 (+ 1047us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.520442 (+ 1143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260810 11:40:47.523029 24238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:47.523310 24238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.525005 24087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.480058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33428 (local address 127.3.240.253:32957)
0810 11:40:47.480463 (+ 405us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.480470 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.488419 (+ 7949us) server_negotiation.cc:407] Connection header received
0810 11:40:47.495116 (+ 6697us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.495127 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.495250 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.495356 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.498522 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.499726 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.502021 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.502664 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.502762 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.508678 (+ 5916us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.508685 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.508690 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.508740 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.516105 (+ 7365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.519021 (+ 2916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.519028 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.519031 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.519226 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.522283 (+ 3057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.522291 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.522296 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.522746 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.523063 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.524403 (+ 1340us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.524657 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255}
I20260810 11:40:47.528920 24238 leader_election.cc:288] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b43b8992772b4b07ae46fd84cbc93e64 (127.3.240.254:46607), 6f4e8e75cc79473897826bb7c0b8f5ad (127.3.240.253:32957)
I20260810 11:40:47.529376 24078 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } attempt: 5
I20260810 11:40:47.531850 23928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44234bf7f3af4bc5a9b991cce73de243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43b8992772b4b07ae46fd84cbc93e64" is_pre_election: true
I20260810 11:40:47.532485 23928 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44234bf7f3af4bc5a9b991cce73de243 in term 0.
I20260810 11:40:47.533627 24118 leader_election.cc:302] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234bf7f3af4bc5a9b991cce73de243, b43b8992772b4b07ae46fd84cbc93e64; no voters:
I20260810 11:40:47.540397 24238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:40:47.540597 24238 raft_consensus.cc:482] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:40:47.540711 24238 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:47.552870 24199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.488671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47360 (local address 127.3.240.252:34237)
0810 11:40:47.488864 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.488870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.504475 (+ 15605us) server_negotiation.cc:407] Connection header received
0810 11:40:47.504805 (+ 330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.504811 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.504910 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.504989 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.507257 (+ 2268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.508450 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.512890 (+ 4440us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.513614 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.513748 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.520924 (+ 7176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.520932 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.520937 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.520997 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.528435 (+ 7438us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.547659 (+ 19224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.547666 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.547670 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.547855 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.549459 (+ 1604us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.549467 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.549470 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.549861 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.550214 (+ 353us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.552226 (+ 2012us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.552493 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
Aug 10 11:40:47 dist-test-slave-drl0 krb5kdc[23839](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362046, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260810 11:40:47.565143 24238 raft_consensus.cc:504] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.569154 23928 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44234bf7f3af4bc5a9b991cce73de243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b43b8992772b4b07ae46fd84cbc93e64"
I20260810 11:40:47.569401 23928 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:47.573169 23928 raft_consensus.cc:2454] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44234bf7f3af4bc5a9b991cce73de243 in term 1.
I20260810 11:40:47.576363 24118 leader_election.cc:302] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44234bf7f3af4bc5a9b991cce73de243, b43b8992772b4b07ae46fd84cbc93e64; no voters:
I20260810 11:40:47.581198 24238 leader_election.cc:288] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [CANDIDATE]: Term 1 election: Requested vote from peers b43b8992772b4b07ae46fd84cbc93e64 (127.3.240.254:46607), 6f4e8e75cc79473897826bb7c0b8f5ad (127.3.240.253:32957)
I20260810 11:40:47.581720 24238 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:47.582188 24238 raft_consensus.cc:686] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [term 1 LEADER]: Becoming Leader. State: Replica: 44234bf7f3af4bc5a9b991cce73de243, State: Running, Role: LEADER
I20260810 11:40:47.584863 24238 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.588264 24086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.530098 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:34237 (local address 127.0.0.1:47364)
0810 11:40:47.530838 (+ 740us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:47.530857 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:47.530980 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:47.535559 (+ 4579us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:47.535568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:47.536245 (+ 677us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:47.536522 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.536538 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.538420 (+ 1882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.538427 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.540076 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:47.540096 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.548259 (+ 8163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.548266 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:47.548537 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.550580 (+ 2043us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:47.550605 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:47.556193 (+ 5588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:47.580990 (+ 24797us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.581000 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.581008 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.585261 (+ 4253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.585853 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:47.585863 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:47.585869 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:47.586253 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:47.587098 (+ 845us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:47.587103 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:47.587338 (+ 235us) client_negotiation.cc:769] Sending connection context
0810 11:40:47.587491 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 11:40:47.587823 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603}
I20260810 11:40:47.590478 24201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:47.533077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.3.240.252:34237)
0810 11:40:47.534791 (+ 1714us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:47.534799 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:47.534831 (+ 32us) server_negotiation.cc:407] Connection header received
0810 11:40:47.534914 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:47.534921 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:47.535070 (+ 149us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:47.535229 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:47.536744 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.538169 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.545449 (+ 7280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:47.546155 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:47.548037 (+ 1882us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:47.556877 (+ 8840us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:47.556884 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:47.556889 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:47.556931 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:47.580642 (+ 23711us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.585490 (+ 4848us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.585499 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.585503 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.585675 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:47.586431 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:47.586436 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:47.586440 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:47.586814 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:47.587008 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:47.589894 (+ 2886us) server_negotiation.cc:299] Negotiation successful
0810 11:40:47.590150 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1524}
I20260810 11:40:47.597800 24078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: Bootstrap starting.
W20260810 11:40:47.601190 24122 tablet.cc:2369] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:40:47.603955 24078 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:47.605377 24078 log.cc:824] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:47.610459 24238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44234bf7f3af4bc5a9b991cce73de243. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.611951 24238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:47.622354 24252 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:47.631307 24078 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: No bootstrap required, opened a new log
I20260810 11:40:47.646629 24078 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.647200 24078 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:47.647403 24078 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f4e8e75cc79473897826bb7c0b8f5ad, State: Initialized, Role: FOLLOWER
I20260810 11:40:47.648106 24078 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } }
I20260810 11:40:47.649788 24255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.653689 24252 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:47.650859 24078 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:47.650923 24035 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44234bf7f3af4bc5a9b991cce73de243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad"
I20260810 11:40:47.662397 24035 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:47.651017 24036 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44234bf7f3af4bc5a9b991cce73de243" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" is_pre_election: true
I20260810 11:40:47.672349 24035 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44234bf7f3af4bc5a9b991cce73de243 in term 1.
I20260810 11:40:47.650806 24255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.703320 24275 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 11:40:47.712602 24035 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [term 1 FOLLOWER]: Refusing update from remote peer 44234bf7f3af4bc5a9b991cce73de243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:47.713740 23928 raft_consensus.cc:1261] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [term 1 FOLLOWER]: Refusing update from remote peer 44234bf7f3af4bc5a9b991cce73de243: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:47.713711 24238 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:47.714733 24248 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:47.710683 24091 master_runner.cc:186] Error getting master registration for 127.3.240.253:32957: OK, instance_id { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" instance_seqno: 1786362046791046 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:40:47.723685 24091 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 11:40:47.731346 24208 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44234bf7f3af4bc5a9b991cce73de243. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.731652 24208 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.733240 24255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44234bf7f3af4bc5a9b991cce73de243. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.733584 24255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.736099 24261 mvcc.cc:205] Tried to move back new op lower bound from 7316938947418509312 to 7316938946993012736. Current Snapshot: MvccSnapshot[applied={T|T < 7316938947418509312}]
I20260810 11:40:47.739506 24252 catalog_manager.cc:1347] Generated new cluster ID: 16c97d6f162340fca149362dea33b51f
I20260810 11:40:47.739612 24252 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:47.740316 24248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.740792 24248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:47.741068 24248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.741377 24248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:47.755731 24208 sys_catalog.cc:455] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.756137 24208 sys_catalog.cc:458] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:40:47.767166 24255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44234bf7f3af4bc5a9b991cce73de243" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b43b8992772b4b07ae46fd84cbc93e64" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46607 } } peers { permanent_uuid: "6f4e8e75cc79473897826bb7c0b8f5ad" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 32957 } } peers { permanent_uuid: "44234bf7f3af4bc5a9b991cce73de243" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 34237 } } }
I20260810 11:40:47.767513 24255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad [sys.catalog]: This master's current role is: FOLLOWER
W20260810 11:40:47.780395 24287 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:40:47.780603 24287 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:40:47.811604 24252 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:40:47.813640 24252 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:47.835193 24252 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243: Generated new TSK 0
I20260810 11:40:47.836694 24252 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:40:48.479243 24221 catalog_manager.cc:1259] Loaded cluster ID: 16c97d6f162340fca149362dea33b51f
I20260810 11:40:48.479377 24221 catalog_manager.cc:1552] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: loading cluster ID for follower catalog manager: success
I20260810 11:40:48.482299 24221 catalog_manager.cc:1574] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: acquiring CA information for follower catalog manager: success
I20260810 11:40:48.484105 24221 catalog_manager.cc:1602] T 00000000000000000000000000000000 P b43b8992772b4b07ae46fd84cbc93e64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:40:48.782739 24287 catalog_manager.cc:1259] Loaded cluster ID: 16c97d6f162340fca149362dea33b51f
I20260810 11:40:48.782856 24287 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: loading cluster ID for follower catalog manager: success
I20260810 11:40:48.785610 24287 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: acquiring CA information for follower catalog manager: success
I20260810 11:40:48.787631 24287 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 6f4e8e75cc79473897826bb7c0b8f5ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:40:48.984035 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 23854
I20260810 11:40:48.985742 23954 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:49.112202 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 23854
I20260810 11:40:49.145210 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 23974
I20260810 11:40:49.146780 24071 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:49.272866 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 23974
I20260810 11:40:49.295611 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:49.294702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46607 (local address 127.0.0.1:41936)
0810 11:40:49.295290 (+ 588us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:49.295429 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":439,"thread_start_us":330,"threads_started":1}
W20260810 11:40:49.296017 24118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46607: connect: Connection refused (error 111)
I20260810 11:40:49.297534 24305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:49.297134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46607 (local address 127.0.0.1:41952)
0810 11:40:49.297285 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:49.297395 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46607: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 11:40:49.297863 24118 consensus_peers.cc:487] T 00000000000000000000000000000000 P 44234bf7f3af4bc5a9b991cce73de243 -> Peer b43b8992772b4b07ae46fd84cbc93e64 (127.3.240.254:46607): Couldn't send request to peer b43b8992772b4b07ae46fd84cbc93e64. Status: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46607: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260810 11:40:49.303624 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 24091
I20260810 11:40:49.305083 24185 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:49.436436 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 24091
2026-08-10T11:40:49Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5384 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:40:49 dist-test-slave-drl0 krb5kdc[24314](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:40:49 dist-test-slave-drl0 krb5kdc[24314](info): set up 2 sockets
Aug 10 11:40:49 dist-test-slave-drl0 krb5kdc[24314](info): commencing operation
krb5kdc: starting...
W20260810 11:40:50.827802 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.329s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:40:51 dist-test-slave-drl0 krb5kdc[24314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:40:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:40:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:40:51.187441 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39941
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:44011
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:51.579977 24332 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:51.580296 24332 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:51.580391 24332 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:51.588019 24332 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:51.588145 24332 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:51.588194 24332 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:51.588246 24332 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:51.588285 24332 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:51.598477 24332 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:44011
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:51.599912 24332 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:51.601219 24332 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:51.614611 24338 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:51.614611 24337 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:51.614832 24343 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:51.615531 24332 server_base.cc:1034] running on GCE node
I20260810 11:40:51.616298 24332 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:51.617798 24332 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:51.619071 24332 hybrid_clock.cc:648] HybridClock initialized: now 1786362051619009 us; error 30 us; skew 500 ppm
I20260810 11:40:51.619460 24332 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:51 dist-test-slave-drl0 krb5kdc[24314](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362051, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:51.629128 24332 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:51.631242 24332 webserver.cc:460] Webserver started at http://127.3.240.254:39013/ using document root <none> and password file <none>
I20260810 11:40:51.631989 24332 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:51.632149 24332 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:51.632488 24332 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:51.635703 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "2b18634a042443cd83aed2b4d3e46658"
format_stamp: "Formatted at 2026-08-10 11:40:51 on dist-test-slave-drl0"
server_key: "e5a642ffc4061b1e696508d52160a834"
server_key_iv: "902616e7176c47415e81728eb928a837"
server_key_version: "encryptionkey@0"
I20260810 11:40:51.636507 24332 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2b18634a042443cd83aed2b4d3e46658"
format_stamp: "Formatted at 2026-08-10 11:40:51 on dist-test-slave-drl0"
server_key: "e5a642ffc4061b1e696508d52160a834"
server_key_iv: "902616e7176c47415e81728eb928a837"
server_key_version: "encryptionkey@0"
I20260810 11:40:51.641961 24332 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:40:51.645989 24354 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:51.647681 24332 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:40:51.647888 24332 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2b18634a042443cd83aed2b4d3e46658"
format_stamp: "Formatted at 2026-08-10 11:40:51 on dist-test-slave-drl0"
server_key: "e5a642ffc4061b1e696508d52160a834"
server_key_iv: "902616e7176c47415e81728eb928a837"
server_key_version: "encryptionkey@0"
I20260810 11:40:51.648111 24332 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:51.671505 24332 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:51.677110 24332 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:51.677493 24332 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:51.701116 24332 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39941
I20260810 11:40:51.701141 24431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39941 every 8 connection(s)
I20260810 11:40:51.702850 24332 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:51.705035 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 24332
I20260810 11:40:51.705360 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:40:51.706425 4035 external_mini_cluster.cc:1428] Setting key cf8c68d5ee2c3134434f22ff0b4a821e
I20260810 11:40:51.708714 24436 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 11:40:51 dist-test-slave-drl0 krb5kdc[24314](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362051, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:51.725286 24436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658: Bootstrap starting.
I20260810 11:40:51.730747 24436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:51.732091 24436 log.cc:824] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:51.735414 24436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658: No bootstrap required, opened a new log
I20260810 11:40:51.736773 24440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:51.709045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47994 (local address 127.3.240.254:39941)
0810 11:40:51.710049 (+ 1004us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:51.710069 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:51.710134 (+ 65us) server_negotiation.cc:407] Connection header received
0810 11:40:51.712761 (+ 2627us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:51.712801 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:51.713586 (+ 785us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:51.714144 (+ 558us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:51.715748 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:51.717852 (+ 2104us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:51.719306 (+ 1454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:51.720401 (+ 1095us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:51.720610 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:51.726090 (+ 5480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:51.726103 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:51.726130 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:51.726191 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:51.731076 (+ 4885us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:51.731998 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:51.732008 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:51.732020 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:51.732306 (+ 286us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:51.732979 (+ 673us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:51.732985 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:51.732987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:51.733828 (+ 841us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:51.734261 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:51.734766 (+ 505us) server_negotiation.cc:299] Negotiation successful
0810 11:40:51.735424 (+ 658us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":708,"thread_start_us":420,"threads_started":1}
I20260810 11:40:51.741585 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:40:51.745306 24436 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } }
I20260810 11:40:51.745716 24436 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:51.745814 24436 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b18634a042443cd83aed2b4d3e46658, State: Initialized, Role: FOLLOWER
I20260810 11:40:51.746789 24436 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } }
I20260810 11:40:51.747242 24443 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:51.747532 24443 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } }
I20260810 11:40:51.748329 24443 leader_election.cc:302] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b18634a042443cd83aed2b4d3e46658; no voters:
I20260810 11:40:51.747221 24436 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:40:51.749370 24436 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:40:51.749403 24443 leader_election.cc:288] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers
I20260810 11:40:51.749547 24436 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:51.751999 24436 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } }
I20260810 11:40:51.752316 24436 leader_election.cc:302] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2b18634a042443cd83aed2b4d3e46658; no voters:
I20260810 11:40:51.752365 24446 raft_consensus.cc:2741] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 11:40:51.752592 24436 leader_election.cc:288] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:40:51.752801 24443 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:51.753728 24443 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [term 1 LEADER]: Becoming Leader. State: Replica: 2b18634a042443cd83aed2b4d3e46658, State: Running, Role: LEADER
I20260810 11:40:51.754421 24443 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } }
I20260810 11:40:51.754873 24436 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:51.757318 24446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2b18634a042443cd83aed2b4d3e46658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } } }
I20260810 11:40:51.757900 24446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:51.758533 24447 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2b18634a042443cd83aed2b4d3e46658. Latest consensus state: current_term: 1 leader_uuid: "2b18634a042443cd83aed2b4d3e46658" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2b18634a042443cd83aed2b4d3e46658" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39941 } } }
I20260810 11:40:51.758903 24447 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:51.759112 24456 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:51.765393 24456 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:51.775919 24456 catalog_manager.cc:1347] Generated new cluster ID: 0076729d5c87474ea396ad5b98b651a6
I20260810 11:40:51.776062 24456 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:51.791221 24456 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:40:51.792968 24456 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:51.810647 24456 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2b18634a042443cd83aed2b4d3e46658: Generated new TSK 0
I20260810 11:40:51.811698 24456 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:40:51.856359 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 24332
I20260810 11:40:51.857852 24427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:51.961129 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 24332
2026-08-10T11:40:51Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2516 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:40:52 dist-test-slave-drl0 krb5kdc[24478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:40:52 dist-test-slave-drl0 krb5kdc[24478](info): set up 2 sockets
Aug 10 11:40:52 dist-test-slave-drl0 krb5kdc[24478](info): commencing operation
krb5kdc: starting...
W20260810 11:40:53.330332 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.000s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:40:53 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:40:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:40:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:40:53.659273 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:54.042021 24504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:54.042364 24504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:54.042532 24504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:54.050308 24504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:54.050465 24504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:54.050576 24504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:54.050675 24504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:54.050760 24504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:54.061280 24504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:54.062724 24504 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:54.064239 24504 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:54.077766 24514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:54.077698 24515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:54.077900 24517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:54.078711 24504 server_base.cc:1034] running on GCE node
I20260810 11:40:54.079609 24504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:54.081389 24504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:54.082687 24504 hybrid_clock.cc:648] HybridClock initialized: now 1786362054082634 us; error 38 us; skew 500 ppm
I20260810 11:40:54.083186 24504 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:54 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362054, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:54.086546 24504 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:54.088619 24504 webserver.cc:460] Webserver started at http://127.3.240.254:35683/ using document root <none> and password file <none>
I20260810 11:40:54.089355 24504 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:54.089495 24504 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:54.089787 24504 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:54.093132 24504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "2dc087d8da5d436da17fecfabaaa53f0"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "5868851d91c75e8fc39289457b01335e"
server_key_iv: "3ec3985be23426395417fe9ad03630be"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.093906 24504 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2dc087d8da5d436da17fecfabaaa53f0"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "5868851d91c75e8fc39289457b01335e"
server_key_iv: "3ec3985be23426395417fe9ad03630be"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.100129 24504 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 11:40:54.104674 24528 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:54.106731 24504 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 11:40:54.107048 24504 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2dc087d8da5d436da17fecfabaaa53f0"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "5868851d91c75e8fc39289457b01335e"
server_key_iv: "3ec3985be23426395417fe9ad03630be"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.107339 24504 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:54.123037 24504 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:54.127710 24504 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:54.128026 24504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:54.149438 24504 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46745
I20260810 11:40:54.149519 24598 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46745 every 8 connection(s)
I20260810 11:40:54.150866 24504 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:54.154883 24599 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:54.155220 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 24504
I20260810 11:40:54.155460 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:40:54.156240 4035 external_mini_cluster.cc:1428] Setting key 7242af37bbed74a5e9b8a36f512b1974
I20260810 11:40:54.170315 24599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap starting.
Aug 10 11:40:54 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:54.174768 24599 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:54.175959 24599 log.cc:824] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:54.180143 24599 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: No bootstrap required, opened a new log
I20260810 11:40:54.185758 24602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:54.158444 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37624 (local address 127.3.240.254:46745)
0810 11:40:54.159736 (+ 1292us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:54.159749 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:54.159802 (+ 53us) server_negotiation.cc:407] Connection header received
0810 11:40:54.161522 (+ 1720us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:54.161543 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:54.161910 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:54.162227 (+ 317us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:54.164370 (+ 2143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.166070 (+ 1700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.168296 (+ 2226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.169448 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.169693 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:54.176238 (+ 6545us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:54.176252 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:54.176277 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:54.176342 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:54.180133 (+ 3791us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:54.181044 (+ 911us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:54.181056 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:54.181067 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:54.181404 (+ 337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:54.181866 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:54.181872 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:54.181874 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:54.182952 (+ 1078us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:54.183419 (+ 467us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:54.183776 (+ 357us) server_negotiation.cc:299] Negotiation successful
0810 11:40:54.184297 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":884,"thread_start_us":453,"threads_started":1}
I20260810 11:40:54.191061 24599 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:54.191434 24599 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:54.191505 24599 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Initialized, Role: FOLLOWER
I20260810 11:40:54.191996 24599 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:54.192198 24599 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:40:54.192307 24599 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:40:54.192435 24599 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:54.194756 24599 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:54.195289 24599 leader_election.cc:302] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dc087d8da5d436da17fecfabaaa53f0; no voters:
I20260810 11:40:54.196034 24605 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:54.195920 24599 leader_election.cc:288] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:40:54.196725 24605 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 LEADER]: Becoming Leader. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Running, Role: LEADER
I20260810 11:40:54.197419 24605 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:54.199255 24599 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:54.200510 24606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dc087d8da5d436da17fecfabaaa53f0. Latest consensus state: current_term: 1 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:40:54.200538 24608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:40:54.200973 24606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:54.200973 24608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:54.202977 24614 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:54.211759 24614 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:54.220703 24614 catalog_manager.cc:1347] Generated new cluster ID: 30a3a6ac95c34da096f60a27574e8397
I20260810 11:40:54.220829 24614 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:54.241746 24614 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:40:54.243135 24614 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:54.254493 24614 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Generated new TSK 0
I20260810 11:40:54.255434 24614 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:40:54.351315 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:54.741319 24642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:54.741608 24642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:54.741767 24642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:54.750478 24642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:54.750593 24642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:54.750792 24642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:40:54.761489 24642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:54.762837 24642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:54.764206 24642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:54.779496 24652 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:54.779582 24650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:54.779448 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:54.780937 24642 server_base.cc:1034] running on GCE node
I20260810 11:40:54.781631 24642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:54.782670 24642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:54.783929 24642 hybrid_clock.cc:648] HybridClock initialized: now 1786362054783855 us; error 62 us; skew 500 ppm
I20260810 11:40:54.784286 24642 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:54 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362054, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:54.787724 24642 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:40:54.789295 24642 webserver.cc:460] Webserver started at http://127.3.240.193:34047/ using document root <none> and password file <none>
I20260810 11:40:54.789999 24642 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:54.790103 24642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:54.790444 24642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:54.793421 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "a707908d645c4e759a561ea752a0d844"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "a5527ab534166b5388dcba9046ac19e0"
server_key_iv: "d24e729a89484d868c050d104d91e05f"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.794196 24642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "a707908d645c4e759a561ea752a0d844"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "a5527ab534166b5388dcba9046ac19e0"
server_key_iv: "d24e729a89484d868c050d104d91e05f"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.800230 24642 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 11:40:54.804559 24663 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:54.806641 24642 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 11:40:54.806944 24642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "a707908d645c4e759a561ea752a0d844"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "a5527ab534166b5388dcba9046ac19e0"
server_key_iv: "d24e729a89484d868c050d104d91e05f"
server_key_version: "encryptionkey@0"
I20260810 11:40:54.807413 24642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:54.837687 24642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:54.842275 24642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:54.842595 24642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:54.843786 24642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:54.845371 24642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:54.845446 24642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:54.845568 24642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:54.845634 24642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:54.884292 24642 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38057
I20260810 11:40:54.884388 24844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38057 every 8 connection(s)
I20260810 11:40:54.885602 24642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:40:54.887745 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 24642
I20260810 11:40:54.887975 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:40:54.889314 4035 external_mini_cluster.cc:1428] Setting key 8f78509f1e3c4179a2f690ba6c8633ca
Aug 10 11:40:54 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362054, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:54.909269 24850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:54.888720 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:59497 (local address 127.3.240.254:46745)
0810 11:40:54.889365 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:54.889374 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:54.890237 (+ 863us) server_negotiation.cc:407] Connection header received
0810 11:40:54.892190 (+ 1953us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:54.892196 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:54.892285 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:54.892405 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:54.893869 (+ 1464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.894614 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.896216 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.896610 (+ 394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.896746 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:54.901668 (+ 4922us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:54.901674 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:54.901678 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:54.901711 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:54.905239 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:54.906213 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:54.906220 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:54.906224 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:54.906373 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:54.906863 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:54.906868 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:54.906871 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:54.907375 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:54.907576 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:54.908487 (+ 911us) server_negotiation.cc:299] Negotiation successful
0810 11:40:54.908877 (+ 390us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":357,"threads_started":1}
I20260810 11:40:54.910364 24851 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:54.889092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:59497)
0810 11:40:54.890063 (+ 971us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:54.890112 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:54.891912 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:54.892633 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:54.892640 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:54.893157 (+ 517us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:54.893702 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:54.893718 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.894783 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.894787 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:54.896065 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:54.896078 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:54.896837 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:54.896844 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:54.897072 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:54.898113 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:54.898150 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:54.901442 (+ 3292us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:54.905443 (+ 4001us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:54.905459 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:54.905485 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:54.906049 (+ 564us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:54.906494 (+ 445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:54.906500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:54.906504 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:54.906726 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:54.907558 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:54.907569 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:54.908031 (+ 462us) client_negotiation.cc:769] Sending connection context
0810 11:40:54.908351 (+ 320us) client_negotiation.cc:240] Negotiation successful
0810 11:40:54.909141 (+ 790us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":443,"threads_started":1}
I20260810 11:40:54.912513 24847 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:54.912956 24847 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:54.914047 24847 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:54.916411 24548 ts_manager.cc:194] Registered new tserver with Master: a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057)
I20260810 11:40:54.918128 24548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:59497
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:40:54.966456 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:55.361634 24856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:55.361932 24856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:55.362033 24856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:55.370316 24856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:55.370440 24856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:55.370563 24856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:40:55.381295 24856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:55.382632 24856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:55.384344 24856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:55.399466 24868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:55.399466 24870 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:55.399679 24873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:55.401036 24856 server_base.cc:1034] running on GCE node
I20260810 11:40:55.401769 24856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:55.402774 24856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:55.404059 24856 hybrid_clock.cc:648] HybridClock initialized: now 1786362055404010 us; error 40 us; skew 500 ppm
I20260810 11:40:55.404421 24856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:55 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362055, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:55.407928 24856 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:40:55.409617 24856 webserver.cc:460] Webserver started at http://127.3.240.194:42007/ using document root <none> and password file <none>
I20260810 11:40:55.410336 24856 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:55.410440 24856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:55.410748 24856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:55.414156 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "b035d45e24344551a34c1b0529a03179"
format_stamp: "Formatted at 2026-08-10 11:40:55 on dist-test-slave-drl0"
server_key: "1da3e2846b03391219507954f748ac0d"
server_key_iv: "c6ef5896961c2e1666287bc36adfd377"
server_key_version: "encryptionkey@0"
I20260810 11:40:55.415025 24856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "b035d45e24344551a34c1b0529a03179"
format_stamp: "Formatted at 2026-08-10 11:40:55 on dist-test-slave-drl0"
server_key: "1da3e2846b03391219507954f748ac0d"
server_key_iv: "c6ef5896961c2e1666287bc36adfd377"
server_key_version: "encryptionkey@0"
I20260810 11:40:55.420385 24856 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.002s
I20260810 11:40:55.424628 24887 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:55.426213 24856 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:40:55.426357 24856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "b035d45e24344551a34c1b0529a03179"
format_stamp: "Formatted at 2026-08-10 11:40:55 on dist-test-slave-drl0"
server_key: "1da3e2846b03391219507954f748ac0d"
server_key_iv: "c6ef5896961c2e1666287bc36adfd377"
server_key_version: "encryptionkey@0"
I20260810 11:40:55.426584 24856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:55.444099 24856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:55.448966 24856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:55.449306 24856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:55.450409 24856 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:55.452004 24856 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:55.452114 24856 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:55.452286 24856 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:55.452373 24856 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:55.493551 24856 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:41943
I20260810 11:40:55.493592 25046 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:41943 every 8 connection(s)
I20260810 11:40:55.495121 24856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:40:55.497116 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 24856
I20260810 11:40:55.497328 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:40:55.497843 4035 external_mini_cluster.cc:1428] Setting key 3789c8ae41291338337a537edd628627
Aug 10 11:40:55 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362055, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:55.523167 25049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:55.498189 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:37333 (local address 127.3.240.254:46745)
0810 11:40:55.498760 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:55.498766 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:55.499799 (+ 1033us) server_negotiation.cc:407] Connection header received
0810 11:40:55.502549 (+ 2750us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:55.502553 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:55.502614 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:55.502721 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:55.504934 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:55.505895 (+ 961us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:55.509276 (+ 3381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:55.509733 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:55.509847 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:55.514791 (+ 4944us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:55.514796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:55.514800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:55.514841 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:55.519202 (+ 4361us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:55.520319 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:55.520327 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:55.520331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:55.520500 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:55.521051 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:55.521057 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:55.521062 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:55.521393 (+ 331us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:55.521579 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:55.522390 (+ 811us) server_negotiation.cc:299] Negotiation successful
0810 11:40:55.522668 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":302,"threads_started":1}
I20260810 11:40:55.524257 25050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:55.498762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:37333)
0810 11:40:55.499627 (+ 865us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:55.499672 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:55.502187 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:55.503020 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:55.503034 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:55.503902 (+ 868us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:55.504705 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:55.504734 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:55.506170 (+ 1436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:55.506181 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:55.509002 (+ 2821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:55.509033 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:55.510070 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:55.510083 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:55.510492 (+ 409us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:55.511481 (+ 989us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:55.511518 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:55.514513 (+ 2995us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:55.519438 (+ 4925us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:55.519451 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:55.519484 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:55.520109 (+ 625us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:55.520653 (+ 544us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:55.520660 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:55.520664 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:55.520895 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:55.521545 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:55.521554 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:55.521946 (+ 392us) client_negotiation.cc:769] Sending connection context
0810 11:40:55.522292 (+ 346us) client_negotiation.cc:240] Negotiation successful
0810 11:40:55.522868 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":357,"threads_started":1}
I20260810 11:40:55.525929 25047 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:55.526286 25047 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:55.527179 25047 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:55.528946 24548 ts_manager.cc:194] Registered new tserver with Master: b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943)
I20260810 11:40:55.530308 24548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:37333
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:40:55.578409 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:55.921170 24847 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
W20260810 11:40:55.967592 25056 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:55.968003 25056 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:55.968185 25056 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:55.977836 25056 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:55.977972 25056 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:55.978142 25056 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:40:55.994841 25056 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:55.996326 25056 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:55.997720 25056 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:56.012955 25074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:56.013103 25067 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:56.013448 25069 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:56.014178 25056 server_base.cc:1034] running on GCE node
I20260810 11:40:56.015002 25056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:56.016057 25056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:56.017340 25056 hybrid_clock.cc:648] HybridClock initialized: now 1786362056017264 us; error 63 us; skew 500 ppm
I20260810 11:40:56.017696 25056 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:56 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362056, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:56.021081 25056 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:40:56.022673 25056 webserver.cc:460] Webserver started at http://127.3.240.195:34361/ using document root <none> and password file <none>
I20260810 11:40:56.023473 25056 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:56.023624 25056 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:56.023942 25056 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:40:56.028266 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "e3f5a39dbc264c64a395c3e18dedc337"
format_stamp: "Formatted at 2026-08-10 11:40:56 on dist-test-slave-drl0"
server_key: "50c14dbfb7c98cfb87544f437bffa675"
server_key_iv: "ccee31a2d20bef1a959f309e5962d5a8"
server_key_version: "encryptionkey@0"
I20260810 11:40:56.029488 25056 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "e3f5a39dbc264c64a395c3e18dedc337"
format_stamp: "Formatted at 2026-08-10 11:40:56 on dist-test-slave-drl0"
server_key: "50c14dbfb7c98cfb87544f437bffa675"
server_key_iv: "ccee31a2d20bef1a959f309e5962d5a8"
server_key_version: "encryptionkey@0"
I20260810 11:40:56.037030 25056 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 11:40:56.041034 25081 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:56.042718 25056 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:40:56.042973 25056 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "e3f5a39dbc264c64a395c3e18dedc337"
format_stamp: "Formatted at 2026-08-10 11:40:56 on dist-test-slave-drl0"
server_key: "50c14dbfb7c98cfb87544f437bffa675"
server_key_iv: "ccee31a2d20bef1a959f309e5962d5a8"
server_key_version: "encryptionkey@0"
I20260810 11:40:56.043205 25056 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:56.056478 25056 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:56.061514 25056 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:56.061944 25056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:56.063181 25056 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:56.064713 25056 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:56.064821 25056 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:56.065068 25056 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:56.065203 25056 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:56.106427 25056 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43739
I20260810 11:40:56.106626 25246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43739 every 8 connection(s)
I20260810 11:40:56.107990 25056 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:40:56.113948 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 25056
I20260810 11:40:56.114177 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:40:56.114791 4035 external_mini_cluster.cc:1428] Setting key 7aeb67959de3a6d1ad7e656951d58c5f
Aug 10 11:40:56 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362056, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:56.132710 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.111288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38925 (local address 127.3.240.254:46745)
0810 11:40:56.112027 (+ 739us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.112033 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.113355 (+ 1322us) server_negotiation.cc:407] Connection header received
0810 11:40:56.116222 (+ 2867us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.116227 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.116361 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.116507 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:56.118486 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.119562 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.121290 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.121790 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.121914 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.125511 (+ 3597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:56.125516 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:56.125519 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:56.125553 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:56.128724 (+ 3171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.129725 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.129734 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.129738 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.129923 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.130661 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.130666 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.130669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.131099 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:56.131308 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.132056 (+ 748us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.132331 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":360,"threads_started":1}
I20260810 11:40:56.133608 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.111660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:38925)
0810 11:40:56.113030 (+ 1370us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.113118 (+ 88us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.115890 (+ 2772us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.116833 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.116848 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.117589 (+ 741us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:56.118185 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.118202 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.119803 (+ 1601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.119807 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.121101 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.121114 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.122050 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.122056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.122262 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.123069 (+ 807us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:56.123094 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:56.125289 (+ 2195us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:56.128931 (+ 3642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:56.128939 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:56.128953 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:56.129605 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:56.130240 (+ 635us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:56.130251 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:56.130255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:56.130505 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:56.131292 (+ 787us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:56.131300 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:56.131642 (+ 342us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.131937 (+ 295us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.132475 (+ 538us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":882,"thread_start_us":460,"threads_started":1}
I20260810 11:40:56.135581 25248 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:56.135941 25248 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:56.136690 25248 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:56.138388 24548 ts_manager.cc:194] Registered new tserver with Master: e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:56.139334 24548 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:38925
I20260810 11:40:56.145155 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:40:56.166534 25252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.151294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37630 (local address 127.3.240.254:46745)
0810 11:40:56.151496 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.151503 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.151885 (+ 382us) server_negotiation.cc:407] Connection header received
0810 11:40:56.152054 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.152060 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.152183 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.152286 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:56.154301 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.155550 (+ 1249us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.157174 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.157747 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.157859 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.160554 (+ 2695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:56.160559 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:56.160562 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:56.160596 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:56.163377 (+ 2781us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.164089 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.164093 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.164096 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.164224 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.164735 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.164742 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.164745 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.165321 (+ 576us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:56.165561 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.166121 (+ 560us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.166297 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260810 11:40:56.176216 24548 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:40:56.183298 24548 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:40:56.222311 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.198382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55138 (local address 127.3.240.194:41943)
0810 11:40:56.211528 (+ 13146us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.211541 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.211586 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:40:56.211722 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.211735 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.212142 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.212421 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.213643 (+ 1222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.215374 (+ 1731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.219098 (+ 3724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.220615 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.220762 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.221019 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.221811 (+ 792us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.221977 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12940,"thread_start_us":12567,"threads_started":1}
I20260810 11:40:56.222379 25271 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.198200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41943 (local address 127.0.0.1:55138)
0810 11:40:56.210889 (+ 12689us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.210971 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.211231 (+ 260us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.212770 (+ 1539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.212778 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.212822 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.213364 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.213381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.215586 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.215597 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.218886 (+ 3289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.218899 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.221188 (+ 2289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.221197 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.221475 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.221535 (+ 60us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.221715 (+ 180us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.221912 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12287,"mutex_wait_us":509,"thread_start_us":16240,"threads_started":1}
I20260810 11:40:56.223825 25269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.198345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39586)
0810 11:40:56.214303 (+ 15958us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.214323 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.214451 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.215503 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.215507 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.215533 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.216089 (+ 556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.216103 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.218690 (+ 2587us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.218695 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.220857 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.220879 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.222986 (+ 2107us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.222992 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.223167 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.223200 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.223286 (+ 86us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.223431 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15825,"mutex_wait_us":4,"thread_start_us":13149,"threads_started":1}
I20260810 11:40:56.225023 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.198781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39586 (local address 127.3.240.193:38057)
0810 11:40:56.214165 (+ 15384us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.214171 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.214426 (+ 255us) server_negotiation.cc:407] Connection header received
0810 11:40:56.214628 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.214638 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.214896 (+ 258us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.215147 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.216781 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.218532 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.221086 (+ 2554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.222671 (+ 1585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.222806 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.224422 (+ 1616us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.224498 (+ 76us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.224673 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15226,"thread_start_us":15054,"threads_started":1}
I20260810 11:40:56.225539 25274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.214946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43739 (local address 127.0.0.1:53104)
0810 11:40:56.216835 (+ 1889us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.216861 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.217039 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.217700 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.217704 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.217732 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.218155 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.218164 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.220345 (+ 2181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.220351 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.222534 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.222551 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.224694 (+ 2143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.224700 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.224892 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.224922 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.225026 (+ 104us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.225170 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1769,"mutex_wait_us":1,"thread_start_us":1577,"threads_started":1}
I20260810 11:40:56.225601 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.215487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53104 (local address 127.3.240.195:43739)
0810 11:40:56.216315 (+ 828us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.216321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.216973 (+ 652us) server_negotiation.cc:407] Connection header received
0810 11:40:56.217227 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.217232 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.217499 (+ 267us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.217715 (+ 216us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.218333 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.220164 (+ 1831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.222717 (+ 2553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.224228 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.224347 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.224557 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.225096 (+ 539us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.225259 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":675,"thread_start_us":482,"threads_started":1}
I20260810 11:40:56.226567 24952 tablet_service.cc:1467] Processing CreateTablet for tablet 05cdbb18546a44f99696c99c183ab72f (DEFAULT_TABLE table=test-table [id=d62d6960123a4988b0865a5111201a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:56.228094 24952 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cdbb18546a44f99696c99c183ab72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:56.228868 24737 tablet_service.cc:1467] Processing CreateTablet for tablet 05cdbb18546a44f99696c99c183ab72f (DEFAULT_TABLE table=test-table [id=d62d6960123a4988b0865a5111201a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:56.229231 25155 tablet_service.cc:1467] Processing CreateTablet for tablet 05cdbb18546a44f99696c99c183ab72f (DEFAULT_TABLE table=test-table [id=d62d6960123a4988b0865a5111201a2b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:56.230525 24737 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cdbb18546a44f99696c99c183ab72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:56.230643 25155 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 05cdbb18546a44f99696c99c183ab72f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:56.241248 25277 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Bootstrap starting.
I20260810 11:40:56.245514 25280 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Bootstrap starting.
I20260810 11:40:56.245903 25277 tablet_bootstrap.cc:654] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:56.247470 25277 log.cc:824] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:56.248338 25281 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Bootstrap starting.
I20260810 11:40:56.249616 25280 tablet_bootstrap.cc:654] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:56.250790 25280 log.cc:824] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:56.252704 25281 tablet_bootstrap.cc:654] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:56.253221 25277 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: No bootstrap required, opened a new log
I20260810 11:40:56.253556 25277 ts_tablet_manager.cc:1397] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260810 11:40:56.253712 25281 log.cc:824] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:56.254101 25280 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: No bootstrap required, opened a new log
I20260810 11:40:56.254379 25280 ts_tablet_manager.cc:1397] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 11:40:56.256242 25281 tablet_bootstrap.cc:492] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: No bootstrap required, opened a new log
I20260810 11:40:56.256512 25281 ts_tablet_manager.cc:1397] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 11:40:56.263541 25281 raft_consensus.cc:348] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.263932 25281 raft_consensus.cc:374] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:56.264029 25281 raft_consensus.cc:729] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a707908d645c4e759a561ea752a0d844, State: Initialized, Role: FOLLOWER
I20260810 11:40:56.264456 25277 raft_consensus.cc:348] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.264667 25281 consensus_queue.cc:260] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.264886 25277 raft_consensus.cc:374] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:56.264984 25277 raft_consensus.cc:729] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b035d45e24344551a34c1b0529a03179, State: Initialized, Role: FOLLOWER
I20260810 11:40:56.265642 25277 consensus_queue.cc:260] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.266999 25281 ts_tablet_manager.cc:1428] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Time spent starting tablet: real 0.010s user 0.004s sys 0.006s
I20260810 11:40:56.266844 25280 raft_consensus.cc:348] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.267544 25280 raft_consensus.cc:374] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:56.267715 25280 raft_consensus.cc:729] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f5a39dbc264c64a395c3e18dedc337, State: Initialized, Role: FOLLOWER
I20260810 11:40:56.268499 25280 consensus_queue.cc:260] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.269426 25047 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
I20260810 11:40:56.270287 25277 ts_tablet_manager.cc:1428] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Time spent starting tablet: real 0.017s user 0.013s sys 0.003s
I20260810 11:40:56.271744 25248 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
I20260810 11:40:56.272297 25280 ts_tablet_manager.cc:1428] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Time spent starting tablet: real 0.018s user 0.009s sys 0.008s
W20260810 11:40:56.360234 25251 tablet.cc:2369] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:40:56.388926 24848 tablet.cc:2369] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:40:56.497660 25048 tablet.cc:2369] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:40:56.531755 25288 raft_consensus.cc:482] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:56.532047 25288 raft_consensus.cc:504] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.533788 25288 leader_election.cc:288] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:56.539849 25289 raft_consensus.cc:482] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:56.540174 25289 raft_consensus.cc:504] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.542835 25289 leader_election.cc:288] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057)
I20260810 11:40:56.543781 25273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.534495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:36789 (local address 127.3.240.193:38057)
0810 11:40:56.534822 (+ 327us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.534830 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.536017 (+ 1187us) server_negotiation.cc:407] Connection header received
0810 11:40:56.536334 (+ 317us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.536339 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.536411 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.536528 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.537180 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.538170 (+ 990us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.540775 (+ 2605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.542292 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.542436 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.542517 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.543345 (+ 828us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.543518 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260810 11:40:56.544770 25291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.534538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.3.240.194:36789)
0810 11:40:56.535890 (+ 1352us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.535907 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.536150 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.536551 (+ 401us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.536555 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.536599 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.537017 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.537027 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.538395 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.538403 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.540550 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.540567 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.542662 (+ 2095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.542668 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.543161 (+ 493us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.543190 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.544040 (+ 850us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.544193 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1224,"mutex_wait_us":291,"spinlock_wait_cycles":34048,"thread_start_us":810,"threads_started":1}
I20260810 11:40:56.545135 25290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.534153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43739 (local address 127.3.240.194:58739)
0810 11:40:56.535206 (+ 1053us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.535224 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.535423 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.535983 (+ 560us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.535987 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.536016 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.536597 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.536614 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.538196 (+ 1582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.538204 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.540301 (+ 2097us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.540316 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.542629 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.542638 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.542877 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.542913 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.544314 (+ 1401us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.544492 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":920,"mutex_wait_us":3,"thread_start_us":613,"threads_started":1}
I20260810 11:40:56.546051 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.534215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58739 (local address 127.3.240.195:43739)
0810 11:40:56.534451 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.534461 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.535334 (+ 873us) server_negotiation.cc:407] Connection header received
0810 11:40:56.535769 (+ 435us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.535777 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.535848 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.535965 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.536884 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.537991 (+ 1107us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.540566 (+ 2575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.542142 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.542340 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.542440 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.543102 (+ 662us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.545772 (+ 2670us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:40:56.546658 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "b035d45e24344551a34c1b0529a03179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a707908d645c4e759a561ea752a0d844" is_pre_election: true
I20260810 11:40:56.547111 25181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "b035d45e24344551a34c1b0529a03179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f5a39dbc264c64a395c3e18dedc337" is_pre_election: true
I20260810 11:40:56.547379 24767 raft_consensus.cc:2454] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b035d45e24344551a34c1b0529a03179 in term 0.
I20260810 11:40:56.547560 25181 raft_consensus.cc:2454] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b035d45e24344551a34c1b0529a03179 in term 0.
I20260810 11:40:56.548372 24888 leader_election.cc:302] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337; no voters:
I20260810 11:40:56.548897 25288 raft_consensus.cc:2781] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:40:56.549065 25288 raft_consensus.cc:482] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:40:56.549202 25288 raft_consensus.cc:3037] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:56.550366 25254 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.542236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41943 (local address 127.3.240.195:44941)
0810 11:40:56.542706 (+ 470us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.542727 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.542834 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.543646 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.543651 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.543677 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.543971 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.543984 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.545549 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.545554 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.547740 (+ 2186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.547754 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.549546 (+ 1792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.549553 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.549760 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.549794 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.549895 (+ 101us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.550044 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":346,"mutex_wait_us":3}
I20260810 11:40:56.550360 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.543087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44941 (local address 127.3.240.194:41943)
0810 11:40:56.543251 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.543258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.543275 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:40:56.543330 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.543336 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.543420 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.543500 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.544143 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.545342 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.547981 (+ 2639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.549361 (+ 1380us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.549549 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.549645 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.549956 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.550096 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 11:40:56.551612 24983 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "e3f5a39dbc264c64a395c3e18dedc337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b035d45e24344551a34c1b0529a03179" is_pre_election: true
I20260810 11:40:56.552551 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.542756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38219 (local address 127.3.240.193:38057)
0810 11:40:56.543695 (+ 939us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.543704 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.544243 (+ 539us) server_negotiation.cc:407] Connection header received
0810 11:40:56.544498 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.544516 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.544596 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.544675 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:56.546457 (+ 1782us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.547780 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.550243 (+ 2463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.551505 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.551714 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.551785 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.552097 (+ 312us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.552234 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":516,"threads_started":1}
I20260810 11:40:56.552596 25294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.542345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.3.240.195:38219)
0810 11:40:56.544148 (+ 1803us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:56.544165 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:56.544352 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:56.545168 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:56.545174 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:56.545205 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:56.545810 (+ 605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.545849 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.548009 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.548016 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.550061 (+ 2045us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:56.550074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.551707 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.551713 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:56.551918 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.551955 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 11:40:56.552070 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 11:40:56.552202 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1685,"spinlock_wait_cycles":22272,"thread_start_us":1663,"threads_started":1}
I20260810 11:40:56.552552 25288 raft_consensus.cc:504] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.553105 24983 raft_consensus.cc:2379] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate e3f5a39dbc264c64a395c3e18dedc337 in current term 1: Already voted for candidate b035d45e24344551a34c1b0529a03179 in this term.
I20260810 11:40:56.553221 24767 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "e3f5a39dbc264c64a395c3e18dedc337" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a707908d645c4e759a561ea752a0d844" is_pre_election: true
I20260810 11:40:56.553335 25288 leader_election.cc:288] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [CANDIDATE]: Term 1 election: Requested vote from peers a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:56.553471 24767 raft_consensus.cc:2454] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e3f5a39dbc264c64a395c3e18dedc337 in term 0.
I20260810 11:40:56.554039 25181 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "b035d45e24344551a34c1b0529a03179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f5a39dbc264c64a395c3e18dedc337"
I20260810 11:40:56.554070 25083 leader_election.cc:302] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a707908d645c4e759a561ea752a0d844, e3f5a39dbc264c64a395c3e18dedc337; no voters:
I20260810 11:40:56.554275 25181 raft_consensus.cc:3037] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:56.554256 24766 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "05cdbb18546a44f99696c99c183ab72f" candidate_uuid: "b035d45e24344551a34c1b0529a03179" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a707908d645c4e759a561ea752a0d844"
I20260810 11:40:56.554522 24766 raft_consensus.cc:3037] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:56.556651 25181 raft_consensus.cc:2454] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b035d45e24344551a34c1b0529a03179 in term 1.
I20260810 11:40:56.556866 25289 raft_consensus.cc:2741] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260810 11:40:56.557051 24766 raft_consensus.cc:2454] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b035d45e24344551a34c1b0529a03179 in term 1.
I20260810 11:40:56.557051 24888 leader_election.cc:302] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337; no voters:
I20260810 11:40:56.557356 25288 raft_consensus.cc:2781] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:56.557837 25288 raft_consensus.cc:686] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 LEADER]: Becoming Leader. State: Replica: b035d45e24344551a34c1b0529a03179, State: Running, Role: LEADER
I20260810 11:40:56.558306 25288 consensus_queue.cc:237] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:56.563509 24547 catalog_manager.cc:5495] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 reported cstate change: term changed from 0 to 1, leader changed from <none> to b035d45e24344551a34c1b0529a03179 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "b035d45e24344551a34c1b0529a03179" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } health_report { overall_health: UNKNOWN } } }
I20260810 11:40:56.594429 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.588281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55144 (local address 127.3.240.194:41943)
0810 11:40:56.588570 (+ 289us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.588574 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.588593 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:40:56.588686 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.588690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.588767 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.588909 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:40:56.589330 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.590115 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.591697 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.592092 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.592195 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.593023 (+ 828us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:40:56.593782 (+ 759us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:40:56.593925 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.594106 (+ 181us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.594180 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":157}
I20260810 11:40:56.604993 24766 raft_consensus.cc:1261] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Refusing update from remote peer b035d45e24344551a34c1b0529a03179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:56.605260 25181 raft_consensus.cc:1261] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Refusing update from remote peer b035d45e24344551a34c1b0529a03179: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:56.606144 25295 consensus_queue.cc:1035] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:56.606510 25288 consensus_queue.cc:1035] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:56.638339 24547 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:56.638665 24547 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:37630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:56.643324 24547 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Sending DeleteTablet for 3 replicas of tablet 05cdbb18546a44f99696c99c183ab72f
I20260810 11:40:56.644588 25155 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cdbb18546a44f99696c99c183ab72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:56 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53104
I20260810 11:40:56.644624 24952 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cdbb18546a44f99696c99c183ab72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:56 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55138
I20260810 11:40:56.644624 24737 tablet_service.cc:1514] Processing DeleteTablet for tablet 05cdbb18546a44f99696c99c183ab72f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:56 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39586
I20260810 11:40:56.645766 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 24642
I20260810 11:40:56.657549 25315 tablet_replica.cc:330] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: stopping tablet replica
I20260810 11:40:56.658238 25315 raft_consensus.cc:2229] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:40:56.659726 25315 raft_consensus.cc:2258] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:56.662231 25314 tablet_replica.cc:330] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: stopping tablet replica
I20260810 11:40:56.662657 25314 raft_consensus.cc:2229] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:56.663196 25314 raft_consensus.cc:2258] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:56.665206 25315 ts_tablet_manager.cc:1905] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:56.665534 25314 ts_tablet_manager.cc:1905] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:56.675385 25314 ts_tablet_manager.cc:1918] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:56.675649 25314 log.cc:1196] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/05cdbb18546a44f99696c99c183ab72f
I20260810 11:40:56.676316 25314 ts_tablet_manager.cc:1939] T 05cdbb18546a44f99696c99c183ab72f P e3f5a39dbc264c64a395c3e18dedc337: Deleting consensus metadata
I20260810 11:40:56.678622 24529 catalog_manager.cc:4841] TS e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): tablet 05cdbb18546a44f99696c99c183ab72f (table test-table [id=d62d6960123a4988b0865a5111201a2b]) successfully deleted
I20260810 11:40:56.680655 25315 ts_tablet_manager.cc:1918] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:56.680884 25315 log.cc:1196] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/05cdbb18546a44f99696c99c183ab72f
I20260810 11:40:56.681344 25316 tablet_replica.cc:330] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: stopping tablet replica
I20260810 11:40:56.681725 25315 ts_tablet_manager.cc:1939] T 05cdbb18546a44f99696c99c183ab72f P b035d45e24344551a34c1b0529a03179: Deleting consensus metadata
I20260810 11:40:56.681900 25316 raft_consensus.cc:2229] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:56.682436 25316 raft_consensus.cc:2258] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:56.683898 24529 catalog_manager.cc:4841] TS b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): tablet 05cdbb18546a44f99696c99c183ab72f (table test-table [id=d62d6960123a4988b0865a5111201a2b]) successfully deleted
I20260810 11:40:56.684604 25316 ts_tablet_manager.cc:1905] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 11:40:56 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:40:56.693120 25316 ts_tablet_manager.cc:1918] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:56.693405 25316 log.cc:1196] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/05cdbb18546a44f99696c99c183ab72f
I20260810 11:40:56.694132 25316 ts_tablet_manager.cc:1939] T 05cdbb18546a44f99696c99c183ab72f P a707908d645c4e759a561ea752a0d844: Deleting consensus metadata
I20260810 11:40:56.696363 25292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.647061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39598 (local address 127.3.240.193:38057)
0810 11:40:56.647263 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.647270 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.653305 (+ 6035us) server_negotiation.cc:407] Connection header received
0810 11:40:56.655116 (+ 1811us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.655128 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.655278 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.655390 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:56.660448 (+ 5058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.661949 (+ 1501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.679032 (+ 17083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.679928 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.680154 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.689578 (+ 9424us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:56.689590 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:56.689605 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:56.689645 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:56.693047 (+ 3402us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.693940 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.693948 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.693950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.694101 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.694565 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.694569 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.694571 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.695029 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:56.695375 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.695694 (+ 319us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.695960 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 11:40:56.696555 24530 catalog_manager.cc:4841] TS a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057): tablet 05cdbb18546a44f99696c99c183ab72f (table test-table [id=d62d6960123a4988b0865a5111201a2b]) successfully deleted
I20260810 11:40:56.697305 24837 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:56.796970 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 24642
I20260810 11:40:56.823032 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 24856
Aug 10 11:40:56 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:40:56.844745 25272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:56.824305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55152 (local address 127.3.240.194:41943)
0810 11:40:56.824527 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:56.824533 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:56.824554 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:40:56.824784 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:56.824789 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:56.824869 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:56.824976 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:56.826983 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.827786 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.829980 (+ 2194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:56.830425 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:56.830578 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:56.836633 (+ 6055us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:56.836649 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:56.836668 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:56.836714 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:56.841184 (+ 4470us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.841988 (+ 804us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.841996 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.841998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.842280 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:56.842757 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:56.842765 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:56.842774 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:56.843333 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:56.843716 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:56.844027 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 11:40:56.844305 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:40:56.845633 25040 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:56.977223 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 24856
I20260810 11:40:57.007154 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 25056
Aug 10 11:40:57 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362053, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:40:57.026823 25276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:57.008108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.3.240.195:43739)
0810 11:40:57.008290 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:57.008294 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:57.008315 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:40:57.008637 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:57.008642 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:57.008718 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:57.008932 (+ 214us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:57.010649 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:57.011622 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:57.013523 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:57.014022 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:57.014126 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:57.019345 (+ 5219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:57.019370 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:57.019397 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:57.019467 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:57.023378 (+ 3911us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:57.024203 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:57.024211 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:57.024214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:57.024398 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:57.024815 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:57.024820 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:57.024822 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:57.025349 (+ 527us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:57.025588 (+ 239us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:57.026217 (+ 629us) server_negotiation.cc:299] Negotiation successful
0810 11:40:57.026404 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:40:57.027817 25239 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:57.125240 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 25056
I20260810 11:40:57.154580 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 24504
I20260810 11:40:57.156317 24589 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:57.273573 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 24504
I20260810 11:40:57.308032 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--webserver_interface=127.3.240.254
--webserver_port=35683
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:40:57.694756 25323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:57.695271 25323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:57.695473 25323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:57.705133 25323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:40:57.705279 25323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:57.705363 25323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:57.705504 25323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:40:57.705587 25323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:40:57.719871 25323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:57.721761 25323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:57.723201 25323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:57.753276 25334 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:57.753553 25337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:57.753535 25332 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:57.755024 25323 server_base.cc:1034] running on GCE node
I20260810 11:40:57.756230 25323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:57.758247 25323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:57.759562 25323 hybrid_clock.cc:648] HybridClock initialized: now 1786362057759489 us; error 59 us; skew 500 ppm
I20260810 11:40:57.760115 25323 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:57 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362057, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:57.764590 25323 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:40:57.766490 25323 webserver.cc:460] Webserver started at http://127.3.240.254:35683/ using document root <none> and password file <none>
I20260810 11:40:57.767364 25323 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:57.767491 25323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:57.774773 25323 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:40:57.779278 25349 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:57.781203 25323 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:40:57.781433 25323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2dc087d8da5d436da17fecfabaaa53f0"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "5868851d91c75e8fc39289457b01335e"
server_key_iv: "3ec3985be23426395417fe9ad03630be"
server_key_version: "encryptionkey@0"
I20260810 11:40:57.782613 25323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:57.815207 25323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:57.819854 25323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:57.820180 25323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:57.844628 25323 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46745
I20260810 11:40:57.844663 25422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46745 every 8 connection(s)
I20260810 11:40:57.846670 25323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:40:57.852219 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 25323
I20260810 11:40:57.853314 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:38057
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=34047
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:40:57.857436 25423 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:40:57.863426 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap starting.
I20260810 11:40:57.885236 25423 log.cc:824] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:57.898315 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:40:57.899019 25423 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap complete.
I20260810 11:40:57.909308 25423 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:57.910787 25423 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Initialized, Role: FOLLOWER
I20260810 11:40:57.911788 25423 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:57.912137 25423 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:40:57.912302 25423 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:40:57.912513 25423 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:40:57.916982 25423 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:57.917614 25423 leader_election.cc:302] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dc087d8da5d436da17fecfabaaa53f0; no voters:
I20260810 11:40:57.918372 25423 leader_election.cc:288] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 11:40:57.918517 25431 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 11:40:57.920018 25431 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 LEADER]: Becoming Leader. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Running, Role: LEADER
I20260810 11:40:57.920641 25431 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:40:57.921084 25423 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:40:57.923688 25433 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:40:57.924362 25433 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:57.924283 25434 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dc087d8da5d436da17fecfabaaa53f0. Latest consensus state: current_term: 2 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:40:57.924875 25434 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:40:57.928696 25437 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:40:57.940203 25437 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:40:57.942233 25437 catalog_manager.cc:1259] Loaded cluster ID: 30a3a6ac95c34da096f60a27574e8397
I20260810 11:40:57.942598 25437 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:40:57.947459 25437 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260810 11:40:57.968176 25437 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260810 11:40:57.978358 25437 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:40:57.980901 25437 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Loaded TSK: 0
I20260810 11:40:57.983525 25437 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 11:40:58.274438 25428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:58.274792 25428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:58.275009 25428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:58.282625 25428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:58.282797 25428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:58.283031 25428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:40:58.293725 25428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:38057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=34047
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:58.295224 25428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:58.296595 25428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:58.313382 25464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:58.313746 25466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:58.314817 25468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:58.315153 25428 server_base.cc:1034] running on GCE node
I20260810 11:40:58.315834 25428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:58.316874 25428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:58.318246 25428 hybrid_clock.cc:648] HybridClock initialized: now 1786362058318158 us; error 43 us; skew 500 ppm
I20260810 11:40:58.318894 25428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:58 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362058, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:58.323140 25428 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:40:58.325834 25428 webserver.cc:460] Webserver started at http://127.3.240.193:34047/ using document root <none> and password file <none>
I20260810 11:40:58.326784 25428 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:58.327143 25428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:58.333503 25428 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260810 11:40:58.336591 25480 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:58.338294 25428 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:40:58.338521 25428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "a707908d645c4e759a561ea752a0d844"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "a5527ab534166b5388dcba9046ac19e0"
server_key_iv: "d24e729a89484d868c050d104d91e05f"
server_key_version: "encryptionkey@0"
I20260810 11:40:58.339640 25428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:58.362807 25428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:58.367731 25428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:58.368029 25428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:58.369076 25428 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:58.370630 25428 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:58.370765 25428 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:58.370900 25428 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:58.371017 25428 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260810 11:40:58.412693 25428 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38057
I20260810 11:40:58.412750 25642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38057 every 8 connection(s)
I20260810 11:40:58.414188 25428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:40:58.419730 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 25428
I20260810 11:40:58.420400 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:41943
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=42007
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:58 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362058, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:58.445106 25650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:58.417995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:57265)
0810 11:40:58.418985 (+ 990us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:58.419034 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:58.421043 (+ 2009us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:58.423981 (+ 2938us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:58.423994 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:58.424658 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:58.425313 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:58.425345 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.427688 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.427696 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:58.428967 (+ 1271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:58.429005 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.430053 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.430061 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:58.430360 (+ 299us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:58.431135 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:58.431164 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:58.433987 (+ 2823us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:58.438835 (+ 4848us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:58.438853 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:58.438881 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:58.440165 (+ 1284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:58.440706 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:58.440715 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:58.440734 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:58.441042 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:58.441933 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:58.441944 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:58.442394 (+ 450us) client_negotiation.cc:769] Sending connection context
0810 11:40:58.442708 (+ 314us) client_negotiation.cc:240] Negotiation successful
0810 11:40:58.443459 (+ 751us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":381,"threads_started":1}
I20260810 11:40:58.445487 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:58.417844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:57265 (local address 127.3.240.254:46745)
0810 11:40:58.418453 (+ 609us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:58.418466 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:58.422054 (+ 3588us) server_negotiation.cc:407] Connection header received
0810 11:40:58.422620 (+ 566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:58.422638 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:58.423351 (+ 713us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:58.423776 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:58.425565 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.427482 (+ 1917us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.429153 (+ 1671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.429792 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.430001 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:58.434353 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:58.434363 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:58.434379 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:58.434429 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:58.438503 (+ 4074us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:58.440346 (+ 1843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:58.440355 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:58.440363 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:58.440550 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:58.441263 (+ 713us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:58.441267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:58.441270 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:58.441771 (+ 501us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:58.441923 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:58.443352 (+ 1429us) server_negotiation.cc:299] Negotiation successful
0810 11:40:58.444020 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":274,"threads_started":1}
I20260810 11:40:58.449139 25644 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:58.449622 25644 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:58.450583 25644 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:58.453348 25371 ts_manager.cc:194] Registered new tserver with Master: a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057)
I20260810 11:40:58.455122 25371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:57265
W20260810 11:40:58.791893 25651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:58.792233 25651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:58.792419 25651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:58.800539 25651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:58.800820 25651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:58.801074 25651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:40:58.812695 25651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:41943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=42007
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:58.814224 25651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:58.815639 25651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:58.831995 25659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:58.831972 25660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:58.832293 25662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:58.833410 25651 server_base.cc:1034] running on GCE node
I20260810 11:40:58.834079 25651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:58.835283 25651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:58.836789 25651 hybrid_clock.cc:648] HybridClock initialized: now 1786362058836750 us; error 36 us; skew 500 ppm
I20260810 11:40:58.837163 25651 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:58 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362058, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:58.841619 25651 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:40:58.844166 25651 webserver.cc:460] Webserver started at http://127.3.240.194:42007/ using document root <none> and password file <none>
I20260810 11:40:58.845124 25651 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:58.845280 25651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:58.851505 25651 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260810 11:40:58.854614 25673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:58.856717 25651 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260810 11:40:58.857034 25651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "b035d45e24344551a34c1b0529a03179"
format_stamp: "Formatted at 2026-08-10 11:40:55 on dist-test-slave-drl0"
server_key: "1da3e2846b03391219507954f748ac0d"
server_key_iv: "c6ef5896961c2e1666287bc36adfd377"
server_key_version: "encryptionkey@0"
I20260810 11:40:58.858336 25651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:58.879308 25651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:58.884471 25651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:58.884775 25651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:58.885857 25651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:58.887614 25651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:58.887773 25651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:58.887933 25651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:58.888036 25651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:58.932331 25651 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:41943
I20260810 11:40:58.932426 25854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:41943 every 8 connection(s)
I20260810 11:40:58.934765 25651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:40:58.941330 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 25651
I20260810 11:40:58.942327 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:43739
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=34361
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:40:58 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362058, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:58.964291 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:58.938217 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:58741 (local address 127.3.240.254:46745)
0810 11:40:58.938448 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:58.938453 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:58.939627 (+ 1174us) server_negotiation.cc:407] Connection header received
0810 11:40:58.941667 (+ 2040us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:58.941675 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:58.941869 (+ 194us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:58.942057 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:58.944689 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.945929 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.948099 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.948923 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.949467 (+ 544us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:58.955029 (+ 5562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:58.955044 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:58.955052 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:58.955117 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:58.959328 (+ 4211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:58.960570 (+ 1242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:58.960580 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:58.960586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:58.960749 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:58.961281 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:58.961289 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:58.961295 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:58.961714 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:58.961887 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:58.963499 (+ 1612us) server_negotiation.cc:299] Negotiation successful
0810 11:40:58.963833 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260810 11:40:58.965500 25858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:58.938617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:58741)
0810 11:40:58.939466 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:58.939504 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:58.941232 (+ 1728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:58.942305 (+ 1073us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:58.942314 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:58.943409 (+ 1095us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:58.944405 (+ 996us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:58.944435 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.946304 (+ 1869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.946316 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:58.947835 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:58.947853 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:58.949923 (+ 2070us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:58.949934 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:58.950304 (+ 370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:58.951281 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:58.951330 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:58.954540 (+ 3210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:58.959577 (+ 5037us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:58.959588 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:58.959612 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:58.960402 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:58.960899 (+ 497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:58.960903 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:58.960905 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:58.961136 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:58.962206 (+ 1070us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:58.962217 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:58.962837 (+ 620us) client_negotiation.cc:769] Sending connection context
0810 11:40:58.963335 (+ 498us) client_negotiation.cc:240] Negotiation successful
0810 11:40:58.963939 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":404,"threads_started":1}
I20260810 11:40:58.967578 25855 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:58.967988 25855 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:58.968828 25855 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:58.970451 25371 ts_manager.cc:194] Registered new tserver with Master: b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943)
I20260810 11:40:58.971450 25371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:58741
W20260810 11:40:59.331696 25859 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:40:59.331996 25859 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:40:59.332154 25859 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:40:59.339427 25859 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:40:59.339540 25859 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:40:59.339712 25859 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:40:59.350739 25859 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=34361
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:40:59.352159 25859 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:40:59.353520 25859 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:40:59.367951 25873 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:59.367980 25869 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:40:59.368119 25868 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:40:59.368671 25859 server_base.cc:1034] running on GCE node
I20260810 11:40:59.369292 25859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:40:59.370424 25859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:40:59.371794 25859 hybrid_clock.cc:648] HybridClock initialized: now 1786362059371704 us; error 100 us; skew 500 ppm
I20260810 11:40:59.372174 25859 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:40:59 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362059, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:40:59.375977 25859 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:40:59.378192 25859 webserver.cc:460] Webserver started at http://127.3.240.195:34361/ using document root <none> and password file <none>
I20260810 11:40:59.379334 25859 fs_manager.cc:359] Metadata directory not provided
I20260810 11:40:59.379535 25859 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:40:59.386391 25859 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260810 11:40:59.389899 25884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:59.391594 25859 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 11:40:59.391789 25859 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "e3f5a39dbc264c64a395c3e18dedc337"
format_stamp: "Formatted at 2026-08-10 11:40:56 on dist-test-slave-drl0"
server_key: "50c14dbfb7c98cfb87544f437bffa675"
server_key_iv: "ccee31a2d20bef1a959f309e5962d5a8"
server_key_version: "encryptionkey@0"
I20260810 11:40:59.392974 25859 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:40:59.405838 25859 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:40:59.410611 25859 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:40:59.410982 25859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:40:59.412233 25859 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:40:59.413734 25859 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:40:59.413856 25859 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:59.413990 25859 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:40:59.414077 25859 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:40:59.455770 25859 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43739
I20260810 11:40:59.455866 26057 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43739 every 8 connection(s)
I20260810 11:40:59.457548 25859 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:40:59.459343 25644 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
I20260810 11:40:59.463488 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 25859
Aug 10 11:40:59 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362059, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:40:59.484630 25649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.460634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47467 (local address 127.3.240.254:46745)
0810 11:40:59.460860 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.460866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.462192 (+ 1326us) server_negotiation.cc:407] Connection header received
0810 11:40:59.464732 (+ 2540us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.464738 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.464843 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.464961 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:59.466665 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.467708 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.470009 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.470763 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.470878 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.475649 (+ 4771us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:59.475654 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:59.475657 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:59.475689 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:59.480056 (+ 4367us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.481045 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.481049 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.481051 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.481164 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.481701 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.481709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.481713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.482369 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:59.482511 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.483655 (+ 1144us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.484010 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":75,"spinlock_wait_cycles":13568}
I20260810 11:40:59.484678 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.464546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37644 (local address 127.3.240.254:46745)
0810 11:40:59.465164 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.465171 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.467078 (+ 1907us) server_negotiation.cc:407] Connection header received
0810 11:40:59.467270 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.467275 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.467345 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.467451 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:59.469560 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.470656 (+ 1096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.472944 (+ 2288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.473412 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.473514 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.476659 (+ 3145us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:59.476667 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:59.476674 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:59.476732 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:59.480667 (+ 3935us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.481701 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.481709 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.481713 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.482082 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.482622 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.482625 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.482627 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.482889 (+ 262us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:59.483364 (+ 475us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.483788 (+ 424us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.484253 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":202,"server-negotiator.queue_time_us":493,"spinlock_wait_cycles":896,"thread_start_us":354,"threads_started":1}
I20260810 11:40:59.485311 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.461222 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:47467)
0810 11:40:59.462036 (+ 814us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.462075 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.464459 (+ 2384us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.465356 (+ 897us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.465364 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.465878 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:40:59.466463 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.466479 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.467876 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.467883 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.469823 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.469840 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.471199 (+ 1359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.471209 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.471517 (+ 308us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.472287 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:40:59.472316 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:40:59.475384 (+ 3068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:40:59.480221 (+ 4837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:59.480228 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:59.480242 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:59.480895 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:59.481275 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:40:59.481282 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:40:59.481289 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:40:59.481551 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:40:59.482530 (+ 979us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:40:59.482539 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:40:59.482976 (+ 437us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.483380 (+ 404us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.483964 (+ 584us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":313,"threads_started":1}
I20260810 11:40:59.486802 26062 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46745
I20260810 11:40:59.487157 26062 heartbeater.cc:464] Registering TS with master...
I20260810 11:40:59.487928 26062 heartbeater.cc:511] Master 127.3.240.254:46745 requested a full tablet report, sending...
I20260810 11:40:59.489370 25371 ts_manager.cc:194] Registered new tserver with Master: e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:59.490247 25371 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:47467
I20260810 11:40:59.500552 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:40:59.508948 26067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.501755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37646 (local address 127.3.240.254:46745)
0810 11:40:59.501971 (+ 216us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.501976 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.502562 (+ 586us) server_negotiation.cc:407] Connection header received
0810 11:40:59.502828 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.502832 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.503072 (+ 240us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.503207 (+ 135us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:40:59.503806 (+ 599us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.504700 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.506225 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.506718 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.506831 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.507620 (+ 789us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:40:59.508265 (+ 645us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:40:59.508387 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.508584 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.508661 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260810 11:40:59.509888 25371 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:37646:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:40:59.518617 25371 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:40:59.546388 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.533392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39610)
0810 11:40:59.535042 (+ 1650us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.535106 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.535378 (+ 272us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.537384 (+ 2006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.537395 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.537457 (+ 62us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:59.538112 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.538145 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.540348 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.540353 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.543046 (+ 2693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.543061 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.545368 (+ 2307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.545376 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.545575 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.545629 (+ 54us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.545764 (+ 135us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.545972 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1214,"thread_start_us":1069,"threads_started":1}
I20260810 11:40:59.547479 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.534049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39610 (local address 127.3.240.193:38057)
0810 11:40:59.536281 (+ 2232us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.536291 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.536329 (+ 38us) server_negotiation.cc:407] Connection header received
0810 11:40:59.536449 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.536458 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.536810 (+ 352us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.537046 (+ 236us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:59.538425 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.540159 (+ 1734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.543242 (+ 3083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.544838 (+ 1596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.544992 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.546891 (+ 1899us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.546983 (+ 92us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.547162 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1867,"thread_start_us":1648,"threads_started":1}
I20260810 11:40:59.550571 26076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.533519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41943 (local address 127.0.0.1:55160)
0810 11:40:59.537784 (+ 4265us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.537813 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.537970 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.539017 (+ 1047us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.539023 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.539062 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:59.539476 (+ 414us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.539487 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.543069 (+ 3582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.543074 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.546433 (+ 3359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.546449 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.548896 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.548902 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.549137 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.549181 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.549292 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.549425 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2450,"thread_start_us":2288,"threads_started":1}
I20260810 11:40:59.551604 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.534075 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55160 (local address 127.3.240.194:41943)
0810 11:40:59.536306 (+ 2231us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.536318 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.538259 (+ 1941us) server_negotiation.cc:407] Connection header received
0810 11:40:59.538404 (+ 145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.538412 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.538686 (+ 274us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.539643 (+ 957us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:59.539860 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.542856 (+ 2996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.546754 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.548422 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.548550 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.548723 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.550960 (+ 2237us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.551226 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2062,"thread_start_us":3153,"threads_started":1}
I20260810 11:40:59.553184 25556 tablet_service.cc:1467] Processing CreateTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 (DEFAULT_TABLE table=test-table [id=abbd660e9eea4335900e3bc44ea24443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:59.555510 25556 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc424bb5cc44d77aaa7db72acb6cbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:59.556752 26075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.536236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43739 (local address 127.0.0.1:53120)
0810 11:40:59.547753 (+ 11517us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.547773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.547877 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.551482 (+ 3605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.551486 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.551507 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:59.551616 (+ 109us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.551622 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.553443 (+ 1821us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.553448 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.554667 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.554679 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.556026 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.556032 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.556196 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.556238 (+ 42us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.556326 (+ 88us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.556468 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11396,"thread_start_us":13756,"threads_started":1}
I20260810 11:40:59.556826 25754 tablet_service.cc:1467] Processing CreateTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 (DEFAULT_TABLE table=test-table [id=abbd660e9eea4335900e3bc44ea24443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:59.557835 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.536917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53120 (local address 127.3.240.195:43739)
0810 11:40:59.549987 (+ 13070us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.549997 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.550032 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:40:59.550133 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.550143 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.550387 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.551247 (+ 860us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:59.551775 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.553282 (+ 1507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.554825 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.555851 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.555993 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.557370 (+ 1377us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.557429 (+ 59us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.557535 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12859,"thread_start_us":13994,"threads_started":1}
I20260810 11:40:59.558590 25754 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc424bb5cc44d77aaa7db72acb6cbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:59.562531 25954 tablet_service.cc:1467] Processing CreateTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 (DEFAULT_TABLE table=test-table [id=abbd660e9eea4335900e3bc44ea24443]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:40:59.564567 25954 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cdc424bb5cc44d77aaa7db72acb6cbc4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:40:59.574529 26085 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Bootstrap starting.
I20260810 11:40:59.577180 26086 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Bootstrap starting.
I20260810 11:40:59.578715 26087 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Bootstrap starting.
I20260810 11:40:59.579478 26085 tablet_bootstrap.cc:654] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:59.580677 26085 log.cc:824] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:59.580847 26086 tablet_bootstrap.cc:654] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:59.582114 26087 tablet_bootstrap.cc:654] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Neither blocks nor log segments found. Creating new log.
I20260810 11:40:59.582078 26086 log.cc:824] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:59.583503 26087 log.cc:824] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Log is configured to *not* fsync() on all Append() calls
I20260810 11:40:59.584930 26085 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: No bootstrap required, opened a new log
I20260810 11:40:59.585227 26085 ts_tablet_manager.cc:1397] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260810 11:40:59.585482 26086 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: No bootstrap required, opened a new log
I20260810 11:40:59.585769 26086 ts_tablet_manager.cc:1397] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260810 11:40:59.586457 26087 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: No bootstrap required, opened a new log
I20260810 11:40:59.586695 26087 ts_tablet_manager.cc:1397] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 11:40:59.593204 26085 raft_consensus.cc:348] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.593604 26085 raft_consensus.cc:374] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:59.593708 26085 raft_consensus.cc:729] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a707908d645c4e759a561ea752a0d844, State: Initialized, Role: FOLLOWER
I20260810 11:40:59.594367 26085 consensus_queue.cc:260] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.596243 26085 ts_tablet_manager.cc:1428] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260810 11:40:59.597301 26087 raft_consensus.cc:348] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.597289 26086 raft_consensus.cc:348] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.598238 26087 raft_consensus.cc:374] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:59.598253 26086 raft_consensus.cc:374] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:40:59.598405 26087 raft_consensus.cc:729] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e3f5a39dbc264c64a395c3e18dedc337, State: Initialized, Role: FOLLOWER
I20260810 11:40:59.598405 26086 raft_consensus.cc:729] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b035d45e24344551a34c1b0529a03179, State: Initialized, Role: FOLLOWER
I20260810 11:40:59.599231 26087 consensus_queue.cc:260] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.599229 26086 consensus_queue.cc:260] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.600445 25855 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
I20260810 11:40:59.601145 26086 ts_tablet_manager.cc:1428] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Time spent starting tablet: real 0.015s user 0.014s sys 0.001s
I20260810 11:40:59.601989 26062 heartbeater.cc:503] Master 127.3.240.254:46745 was elected leader, sending a full tablet report...
I20260810 11:40:59.602821 26087 ts_tablet_manager.cc:1428] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Time spent starting tablet: real 0.016s user 0.012s sys 0.004s
W20260810 11:40:59.667045 25647 tablet.cc:2369] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:40:59.687893 25857 tablet.cc:2369] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:40:59.709766 26063 tablet.cc:2369] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:40:59.751593 26091 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:40:59.752107 26091 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.754050 26091 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:59.762907 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.754658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56027 (local address 127.3.240.194:41943)
0810 11:40:59.754895 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.754901 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.755539 (+ 638us) server_negotiation.cc:407] Connection header received
0810 11:40:59.755721 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.755729 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.755867 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.756034 (+ 167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:59.757244 (+ 1210us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.758147 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.760655 (+ 2508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.761719 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.761904 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.761984 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.762455 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.762604 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 11:40:59.763242 26094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.754431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41943 (local address 127.3.240.193:56027)
0810 11:40:59.755303 (+ 872us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.755329 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.755503 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.756295 (+ 792us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.756303 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.756350 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:59.757015 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.757027 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.758321 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.758326 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.760471 (+ 2145us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.760488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.762004 (+ 1516us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.762010 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.762205 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.762242 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.762418 (+ 176us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.762545 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":729,"thread_start_us":530,"threads_started":1}
I20260810 11:40:59.764117 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.755867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:49257 (local address 127.3.240.195:43739)
0810 11:40:59.756167 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.756174 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.757015 (+ 841us) server_negotiation.cc:407] Connection header received
0810 11:40:59.757244 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.757249 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.757344 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.757481 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:40:59.758195 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.758956 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.761176 (+ 2220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.762745 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.762906 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.763016 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.763735 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.763861 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 11:40:59.764225 26096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.755464 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43739 (local address 127.3.240.193:49257)
0810 11:40:59.756807 (+ 1343us) negotiation.cc:107] Waiting for socket to connect
0810 11:40:59.756839 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 11:40:59.757027 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:40:59.757633 (+ 606us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:40:59.757637 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:40:59.757667 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:40:59.758024 (+ 357us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.758033 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.759138 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.759144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.761016 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:40:59.761027 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.763308 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.763315 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:40:59.763554 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.763590 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:40:59.763705 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 11:40:59.763842 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1178,"mutex_wait_us":3,"thread_start_us":746,"threads_started":1}
I20260810 11:40:59.764959 25781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc424bb5cc44d77aaa7db72acb6cbc4" candidate_uuid: "a707908d645c4e759a561ea752a0d844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b035d45e24344551a34c1b0529a03179" is_pre_election: true
I20260810 11:40:59.765528 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc424bb5cc44d77aaa7db72acb6cbc4" candidate_uuid: "a707908d645c4e759a561ea752a0d844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f5a39dbc264c64a395c3e18dedc337" is_pre_election: true
I20260810 11:40:59.765655 25781 raft_consensus.cc:2454] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a707908d645c4e759a561ea752a0d844 in term 0.
I20260810 11:40:59.766166 25982 raft_consensus.cc:2454] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a707908d645c4e759a561ea752a0d844 in term 0.
I20260810 11:40:59.766613 25481 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a707908d645c4e759a561ea752a0d844, b035d45e24344551a34c1b0529a03179; no voters:
I20260810 11:40:59.767211 26091 raft_consensus.cc:2781] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:40:59.767431 26091 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:40:59.767561 26091 raft_consensus.cc:3037] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:59.769887 26091 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.770506 26091 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 1 election: Requested vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
I20260810 11:40:59.771167 25781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc424bb5cc44d77aaa7db72acb6cbc4" candidate_uuid: "a707908d645c4e759a561ea752a0d844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b035d45e24344551a34c1b0529a03179"
I20260810 11:40:59.771401 25781 raft_consensus.cc:3037] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:59.771452 25982 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cdc424bb5cc44d77aaa7db72acb6cbc4" candidate_uuid: "a707908d645c4e759a561ea752a0d844" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e3f5a39dbc264c64a395c3e18dedc337"
I20260810 11:40:59.771704 25982 raft_consensus.cc:3037] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:40:59.774169 25781 raft_consensus.cc:2454] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a707908d645c4e759a561ea752a0d844 in term 1.
I20260810 11:40:59.774521 25982 raft_consensus.cc:2454] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a707908d645c4e759a561ea752a0d844 in term 1.
I20260810 11:40:59.774628 25481 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a707908d645c4e759a561ea752a0d844, b035d45e24344551a34c1b0529a03179; no voters:
I20260810 11:40:59.775099 26091 raft_consensus.cc:2781] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:40:59.775539 26091 raft_consensus.cc:686] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 LEADER]: Becoming Leader. State: Replica: a707908d645c4e759a561ea752a0d844, State: Running, Role: LEADER
I20260810 11:40:59.776086 26091 consensus_queue.cc:237] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:40:59.781639 25371 catalog_manager.cc:5495] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 reported cstate change: term changed from 0 to 1, leader changed from <none> to a707908d645c4e759a561ea752a0d844 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "a707908d645c4e759a561ea752a0d844" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } health_report { overall_health: UNKNOWN } } }
I20260810 11:40:59.845019 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.838880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39622 (local address 127.3.240.193:38057)
0810 11:40:59.839116 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.839120 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.839261 (+ 141us) server_negotiation.cc:407] Connection header received
0810 11:40:59.839433 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.839437 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.839532 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.839674 (+ 142us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:40:59.840138 (+ 464us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.840907 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.842461 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.842864 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.843058 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.843559 (+ 501us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:40:59.844167 (+ 608us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:40:59.844320 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.844653 (+ 333us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.844731 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:40:59.854179 25982 raft_consensus.cc:1261] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Refusing update from remote peer a707908d645c4e759a561ea752a0d844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:59.854506 25781 raft_consensus.cc:1261] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Refusing update from remote peer a707908d645c4e759a561ea752a0d844: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:40:59.855340 26097 consensus_queue.cc:1035] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:59.855813 26091 consensus_queue.cc:1035] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:40:59.888942 25370 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:37646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:59.889377 25370 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:37646:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:40:59.893231 25370 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Sending DeleteTablet for 3 replicas of tablet cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:40:59.894443 25556 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:59 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39610
I20260810 11:40:59.894721 25754 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:59 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55160
I20260810 11:40:59.894948 25954 tablet_service.cc:1514] Processing DeleteTablet for tablet cdc424bb5cc44d77aaa7db72acb6cbc4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:40:59 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:53120
I20260810 11:40:59.896343 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 25428
I20260810 11:40:59.907523 26121 tablet_replica.cc:330] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: stopping tablet replica
I20260810 11:40:59.907927 26122 tablet_replica.cc:330] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: stopping tablet replica
I20260810 11:40:59.908419 26121 raft_consensus.cc:2229] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:59.908470 26122 raft_consensus.cc:2229] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:40:59.909073 26121 raft_consensus.cc:2258] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:59.909096 26122 raft_consensus.cc:2258] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:59.910526 26120 tablet_replica.cc:330] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: stopping tablet replica
I20260810 11:40:59.910974 26120 raft_consensus.cc:2229] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:40:59.911643 26121 ts_tablet_manager.cc:1905] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:59.911736 26120 raft_consensus.cc:2258] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:40:59.913350 26122 ts_tablet_manager.cc:1905] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:40:59.919961 26121 ts_tablet_manager.cc:1918] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:59.920174 26121 log.cc:1196] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:40:59.920805 26121 ts_tablet_manager.cc:1939] T cdc424bb5cc44d77aaa7db72acb6cbc4 P b035d45e24344551a34c1b0529a03179: Deleting consensus metadata
I20260810 11:40:59.920832 26122 ts_tablet_manager.cc:1918] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:40:59.921028 26122 log.cc:1196] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:40:59.921756 26077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:40:59.897495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39630 (local address 127.3.240.193:38057)
0810 11:40:59.897678 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 11:40:59.897684 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:40:59.897719 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:40:59.897789 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:40:59.897794 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:40:59.897894 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:40:59.898040 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:40:59.901037 (+ 2997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.902801 (+ 1764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.905763 (+ 2962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:40:59.907164 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:40:59.907279 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:40:59.912809 (+ 5530us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:40:59.912835 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:40:59.912854 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:40:59.912906 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:40:59.917227 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.918175 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.918184 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.918188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.918349 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:40:59.918816 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:40:59.918822 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:40:59.918826 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:40:59.919347 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:40:59.919650 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0810 11:40:59.920978 (+ 1328us) server_negotiation.cc:299] Negotiation successful
0810 11:40:59.921319 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 11:40:59.922567 25636 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:40:59.923031 25350 catalog_manager.cc:4841] TS b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): tablet cdc424bb5cc44d77aaa7db72acb6cbc4 (table test-table [id=abbd660e9eea4335900e3bc44ea24443]) successfully deleted
I20260810 11:40:59.924379 26122 ts_tablet_manager.cc:1939] T cdc424bb5cc44d77aaa7db72acb6cbc4 P e3f5a39dbc264c64a395c3e18dedc337: Deleting consensus metadata
I20260810 11:40:59.926189 25350 catalog_manager.cc:4841] TS e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): tablet cdc424bb5cc44d77aaa7db72acb6cbc4 (table test-table [id=abbd660e9eea4335900e3bc44ea24443]) successfully deleted
I20260810 11:40:59.923185 26120 ts_tablet_manager.cc:1905] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:00.042279 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 25428
I20260810 11:41:00.071156 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 25651
W20260810 11:41:00.071074 25351 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:38057 (error 108)
I20260810 11:41:00.073493 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.072724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39640)
0810 11:41:00.073144 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.073318 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
W20260810 11:41:00.073737 25351 catalog_manager.cc:4584] TS a707908d645c4e759a561ea752a0d844 (127.3.240.193:38057): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet cdc424bb5cc44d77aaa7db72acb6cbc4: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
I20260810 11:41:00.088354 26081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.072102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55176 (local address 127.3.240.194:41943)
0810 11:41:00.072314 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:00.072320 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:00.072408 (+ 88us) server_negotiation.cc:407] Connection header received
0810 11:41:00.072593 (+ 185us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:00.072598 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:00.072684 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:00.072772 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:00.075151 (+ 2379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:00.076122 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:00.077600 (+ 1478us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:00.078020 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:00.078135 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:00.081285 (+ 3150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:00.081306 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:00.081329 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:00.081389 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:00.085227 (+ 3838us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:00.085977 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:00.085984 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:00.085987 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:00.086143 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:00.086612 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:00.086616 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:00.086618 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:00.087171 (+ 553us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:00.087405 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:00.087755 (+ 350us) server_negotiation.cc:299] Negotiation successful
0810 11:41:00.087990 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 11:41:00.089152 25847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:00.127601 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.127135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39646)
0810 11:41:00.127361 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.127458 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 11:41:00.128579 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.128233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39660)
0810 11:41:00.128399 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.128463 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260810 11:41:00.187418 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 25651
I20260810 11:41:00.198637 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.198174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39666)
0810 11:41:00.198381 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.198467 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 11:41:00.199640 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.199288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39676)
0810 11:41:00.199447 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.199511 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20260810 11:41:00.218731 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 25859
I20260810 11:41:00.236145 26084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.219627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53122 (local address 127.3.240.195:43739)
0810 11:41:00.219796 (+ 169us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:00.219801 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:00.219821 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:41:00.219985 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:00.219990 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:00.220063 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:00.220186 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:00.222316 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:00.223247 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:00.225173 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:00.225563 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:00.225664 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:00.228827 (+ 3163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:00.228849 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:00.228875 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:00.228961 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:00.232836 (+ 3875us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:00.233651 (+ 815us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:00.233661 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:00.233664 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:00.233848 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:00.234456 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:00.234460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:00.234463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:00.234869 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:00.235206 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:00.235685 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 11:41:00.235863 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20260810 11:41:00.236873 26052 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:00.281168 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.280381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39690)
0810 11:41:00.280770 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.280921 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":162}
I20260810 11:41:00.282735 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.282121 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39702)
0810 11:41:00.282422 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.282524 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260810 11:41:00.339809 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 25859
I20260810 11:41:00.364034 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 25323
I20260810 11:41:00.365830 25417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:00.485921 26128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:00.484979 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38057 (local address 127.0.0.1:39718)
0810 11:41:00.485347 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:00.485678 (+ 331us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:38057: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":201}
I20260810 11:41:00.486172 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 25323
I20260810 11:41:00.520895 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--webserver_interface=127.3.240.254
--webserver_port=35683
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46745
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:41:00.900444 26133 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:00.900876 26133 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:00.901126 26133 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:00.909409 26133 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:41:00.909576 26133 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:00.909629 26133 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:00.909785 26133 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:41:00.909854 26133 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:41:00.921049 26133 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46745
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46745
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=35683
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:00.923219 26133 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:00.925118 26133 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:00.944733 26141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:00.944873 26138 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:00.945274 26143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:00.945879 26133 server_base.cc:1034] running on GCE node
I20260810 11:41:00.946830 26133 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:00.948637 26133 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:00.949888 26133 hybrid_clock.cc:648] HybridClock initialized: now 1786362060949818 us; error 60 us; skew 500 ppm
I20260810 11:41:00.950260 26133 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:00 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362060, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:00.954284 26133 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:41:00.956255 26133 webserver.cc:460] Webserver started at http://127.3.240.254:35683/ using document root <none> and password file <none>
I20260810 11:41:00.956969 26133 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:00.957078 26133 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:00.962651 26133 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:41:00.968276 26151 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:00.970255 26133 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:41:00.970590 26133 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2dc087d8da5d436da17fecfabaaa53f0"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "5868851d91c75e8fc39289457b01335e"
server_key_iv: "3ec3985be23426395417fe9ad03630be"
server_key_version: "encryptionkey@0"
I20260810 11:41:00.971942 26133 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:00.992405 26133 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:00.999352 26133 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:00.999696 26133 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:01.027108 26133 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46745
I20260810 11:41:01.027160 26228 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46745 every 8 connection(s)
I20260810 11:41:01.028703 26133 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:41:01.036286 26229 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:41:01.036860 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 26133
I20260810 11:41:01.038146 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:38057
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=34047
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:41:01.042023 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap starting.
I20260810 11:41:01.071447 26229 log.cc:824] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:01.090540 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:41:01.091332 26229 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0: Bootstrap complete.
I20260810 11:41:01.103260 26229 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:41:01.104475 26229 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Initialized, Role: FOLLOWER
I20260810 11:41:01.105623 26229 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:41:01.105975 26229 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:41:01.106103 26229 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:41:01.106341 26229 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 2 FOLLOWER]: Advancing to term 3
I20260810 11:41:01.111517 26229 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:41:01.112262 26229 leader_election.cc:302] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2dc087d8da5d436da17fecfabaaa53f0; no voters:
I20260810 11:41:01.113139 26229 leader_election.cc:288] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 11:41:01.113500 26236 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 11:41:01.115345 26236 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [term 3 LEADER]: Becoming Leader. State: Replica: 2dc087d8da5d436da17fecfabaaa53f0, State: Running, Role: LEADER
I20260810 11:41:01.116397 26236 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } }
I20260810 11:41:01.116866 26229 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:41:01.119822 26238 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:41:01.120391 26238 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:01.124521 26239 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2dc087d8da5d436da17fecfabaaa53f0. Latest consensus state: current_term: 3 leader_uuid: "2dc087d8da5d436da17fecfabaaa53f0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2dc087d8da5d436da17fecfabaaa53f0" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46745 } } }
I20260810 11:41:01.125252 26239 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2dc087d8da5d436da17fecfabaaa53f0 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:01.126897 26245 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:41:01.140226 26245 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:41:01.142333 26245 catalog_manager.cc:1259] Loaded cluster ID: 30a3a6ac95c34da096f60a27574e8397
I20260810 11:41:01.142463 26245 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260810 11:41:01.145303 26245 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7eff37a899ad google::LogMessage::Fail() at ??:0
@ 0x7eff37a8c88a google::LogMessage::SendToLog() at ??:0
@ 0x7eff37a894de google::LogMessage::Flush() at ??:0
@ 0x7eff37a8aa19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7eff4d4bc512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
I20260810 11:41:01.182452 26266 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
@ 0x7eff4d4b9976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7eff4d541610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7eff4d541439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff3a6d5345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7eff3a6d891c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7eff3a6d8799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7eff3a6b3eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7eff3d98d6db start_thread at ??:0
@ 0x7eff3692361f clone at ??:0
W20260810 11:41:01.453416 26234 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:01.453768 26234 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:01.453874 26234 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:01.461134 26234 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:01.461246 26234 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:01.461361 26234 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:41:01.471323 26234 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:38057
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=34047
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:01.472553 26234 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:01.473896 26234 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:01.487861 26293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:01.487986 26290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:01.488446 26289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:01.489285 26234 server_base.cc:1034] running on GCE node
I20260810 11:41:01.489923 26234 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:01.491376 26234 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:01.492664 26234 hybrid_clock.cc:648] HybridClock initialized: now 1786362061492593 us; error 72 us; skew 500 ppm
I20260810 11:41:01.492973 26234 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:01 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:01.497016 26234 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:41:01.499445 26234 webserver.cc:460] Webserver started at http://127.3.240.193:34047/ using document root <none> and password file <none>
I20260810 11:41:01.500217 26234 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:01.500363 26234 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:01.506417 26234 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260810 11:41:01.509850 26300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:01.511502 26234 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 11:41:01.511699 26234 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "a707908d645c4e759a561ea752a0d844"
format_stamp: "Formatted at 2026-08-10 11:40:54 on dist-test-slave-drl0"
server_key: "a5527ab534166b5388dcba9046ac19e0"
server_key_iv: "d24e729a89484d868c050d104d91e05f"
server_key_version: "encryptionkey@0"
I20260810 11:41:01.512965 26234 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:01.530226 26234 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:01.535291 26234 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:01.535636 26234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:01.537453 26234 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:01.540361 26308 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260810 11:41:01.548619 26234 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260810 11:41:01.548808 26234 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.009s user 0.001s sys 0.000s
I20260810 11:41:01.548988 26234 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260810 11:41:01.552759 26234 ts_tablet_manager.cc:610] Registered 1 tablets
I20260810 11:41:01.552855 26234 ts_tablet_manager.cc:589] Time spent register tablets: real 0.004s user 0.000s sys 0.002s
I20260810 11:41:01.553251 26308 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Bootstrap starting.
I20260810 11:41:01.562057 26308 log.cc:824] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:01.577123 26308 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:41:01.578482 26308 tablet_bootstrap.cc:492] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Bootstrap complete.
I20260810 11:41:01.579155 26308 ts_tablet_manager.cc:1397] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Time spent bootstrapping tablet: real 0.026s user 0.020s sys 0.005s
I20260810 11:41:01.590607 26308 raft_consensus.cc:348] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:01.591990 26308 raft_consensus.cc:729] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a707908d645c4e759a561ea752a0d844, State: Initialized, Role: FOLLOWER
I20260810 11:41:01.593148 26308 consensus_queue.cc:260] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:01.595242 26308 ts_tablet_manager.cc:1428] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 11:41:01.602139 26234 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38057
I20260810 11:41:01.602216 26471 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38057 every 8 connection(s)
I20260810 11:41:01.603472 26234 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:41:01.607174 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 26234
I20260810 11:41:01.608062 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:41943
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=42007
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:41:01.609355 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:01.606956 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:57387)
0810 11:41:01.607894 (+ 938us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:01.608336 (+ 442us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":669,"thread_start_us":408,"threads_started":1}
W20260810 11:41:01.610141 26472 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46745 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46745: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
I20260810 11:41:01.611419 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:01.610837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:41851)
0810 11:41:01.611162 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:01.611260 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":197}
I20260810 11:41:01.612433 26474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:01.611946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:47185)
0810 11:41:01.612158 (+ 212us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:01.612261 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 11:41:01.612831 26472 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:41:01.967630 26475 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:01.967932 26475 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:01.968081 26475 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:01.976227 26475 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:01.976341 26475 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:01.976526 26475 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:41:01.986548 26475 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:41943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=42007
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:01.987946 26475 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:01.989310 26475 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:02.003968 26487 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:02.004106 26483 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:02.003901 26484 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:02.005985 26475 server_base.cc:1034] running on GCE node
I20260810 11:41:02.007020 26475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:02.008383 26475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:02.009663 26475 hybrid_clock.cc:648] HybridClock initialized: now 1786362062009574 us; error 69 us; skew 500 ppm
I20260810 11:41:02.010110 26475 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:02 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:02.017385 26475 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:41:02.019078 26475 webserver.cc:460] Webserver started at http://127.3.240.194:42007/ using document root <none> and password file <none>
I20260810 11:41:02.019904 26475 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:02.020018 26475 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:02.025619 26475 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:41:02.028533 26495 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:02.030094 26475 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:41:02.030309 26475 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "b035d45e24344551a34c1b0529a03179"
format_stamp: "Formatted at 2026-08-10 11:40:55 on dist-test-slave-drl0"
server_key: "1da3e2846b03391219507954f748ac0d"
server_key_iv: "c6ef5896961c2e1666287bc36adfd377"
server_key_version: "encryptionkey@0"
I20260810 11:41:02.031419 26475 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:02.066852 26475 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:02.071529 26475 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:02.071825 26475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:02.072865 26475 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:02.074261 26475 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:02.074398 26475 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:02.074529 26475 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:02.074620 26475 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:02.116453 26475 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:41943
I20260810 11:41:02.116514 26657 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:41943 every 8 connection(s)
I20260810 11:41:02.118000 26475 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:41:02.123363 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.121160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:39101)
0810 11:41:02.122064 (+ 904us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.122355 (+ 291us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":326,"threads_started":1}
W20260810 11:41:02.124159 26658 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46745 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46745: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
I20260810 11:41:02.124563 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 26475
I20260810 11:41:02.125486 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.124772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:48987)
0810 11:41:02.125088 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.125248 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 11:41:02.125818 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:43739
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:46745
--webserver_port=34361
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:45987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:41:02.127019 26661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.126425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:41359)
0810 11:41:02.126663 (+ 238us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.126791 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260810 11:41:02.127395 26658 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:41:02.499910 26663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:02.500231 26663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:02.500398 26663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:02.508399 26663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:02.508512 26663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:02.508684 26663 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:41:02.519016 26663 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:45987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:43739
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=34361
--tserver_master_addrs=127.3.240.254:46745
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:02.520568 26663 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:02.521972 26663 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:02.536868 26669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:02.537137 26671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:02.538337 26668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:02.538362 26663 server_base.cc:1034] running on GCE node
I20260810 11:41:02.539256 26663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:02.540388 26663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:02.541654 26663 hybrid_clock.cc:648] HybridClock initialized: now 1786362062541593 us; error 49 us; skew 500 ppm
I20260810 11:41:02.541965 26663 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:02 dist-test-slave-drl0 krb5kdc[24478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362062, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:02.545409 26663 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:41:02.547233 26663 webserver.cc:460] Webserver started at http://127.3.240.195:34361/ using document root <none> and password file <none>
I20260810 11:41:02.547957 26663 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:02.548121 26663 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:02.553467 26663 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260810 11:41:02.556695 26682 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:02.558174 26663 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:41:02.558323 26663 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "e3f5a39dbc264c64a395c3e18dedc337"
format_stamp: "Formatted at 2026-08-10 11:40:56 on dist-test-slave-drl0"
server_key: "50c14dbfb7c98cfb87544f437bffa675"
server_key_iv: "ccee31a2d20bef1a959f309e5962d5a8"
server_key_version: "encryptionkey@0"
I20260810 11:41:02.559535 26663 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:02.586498 26663 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:02.591459 26663 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:02.591768 26663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:02.592981 26663 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:02.594623 26663 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:02.594762 26663 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:02.594923 26663 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:02.595034 26663 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 11:41:02.615341 26759 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.613549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:58765)
0810 11:41:02.614784 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.614948 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1052,"thread_start_us":896,"threads_started":1}
I20260810 11:41:02.639257 26663 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43739
I20260810 11:41:02.639348 26844 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43739 every 8 connection(s)
I20260810 11:41:02.640784 26663 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:41:02.646330 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.644036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:50083)
0810 11:41:02.644951 (+ 915us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.645230 (+ 279us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":322,"threads_started":1}
W20260810 11:41:02.647456 26846 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:46745 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:46745: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
I20260810 11:41:02.648522 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.647962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:39489)
0810 11:41:02.648218 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.648339 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260810 11:41:02.649907 26848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:02.649504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:39293)
0810 11:41:02.649697 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:02.649785 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260810 11:41:02.650282 26846 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 11:41:02.650743 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 26663
I20260810 11:41:03.128975 26850 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.128024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:52429)
0810 11:41:03.128606 (+ 582us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:03.128710 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":461,"thread_start_us":318,"threads_started":1}
I20260810 11:41:03.378450 26851 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:03.378762 26851 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:03.380605 26851 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
Aug 10 11:41:03 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
Aug 10 11:41:03 dist-test-slave-drl0 krb5kdc[24478](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362061, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:41:03.410081 26852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.380865 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:41943 (local address 127.3.240.193:55333)
0810 11:41:03.381441 (+ 576us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:03.381484 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:03.382168 (+ 684us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:03.386061 (+ 3893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:03.386077 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:03.387660 (+ 1583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:03.389214 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:03.389246 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.391834 (+ 2588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.391840 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:03.394377 (+ 2537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:03.394394 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.395984 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.395991 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:03.396482 (+ 491us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:03.397533 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:03.397575 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:03.401082 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:03.404743 (+ 3661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:03.404752 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:03.404767 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:03.405438 (+ 671us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:03.406058 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:03.406065 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:03.406069 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:03.406367 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:03.407555 (+ 1188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:03.407564 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:03.408194 (+ 630us) client_negotiation.cc:769] Sending connection context
0810 11:41:03.408655 (+ 461us) client_negotiation.cc:240] Negotiation successful
0810 11:41:03.409571 (+ 916us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":427,"mutex_wait_us":608,"spinlock_wait_cycles":42112,"thread_start_us":276,"threads_started":1}
I20260810 11:41:03.410298 26853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.381369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:55333 (local address 127.3.240.194:41943)
0810 11:41:03.382305 (+ 936us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:03.382320 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:03.382371 (+ 51us) server_negotiation.cc:407] Connection header received
0810 11:41:03.384765 (+ 2394us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:03.384778 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:03.385249 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:03.385615 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:03.389482 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.391539 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.394565 (+ 3026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.395597 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.395792 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:03.401420 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:03.401436 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:03.401458 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:03.401524 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:03.404570 (+ 3046us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:03.405635 (+ 1065us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:03.405643 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:03.405658 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:03.405839 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:03.406514 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:03.406518 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:03.406520 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:03.407235 (+ 715us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:03.407493 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:03.409108 (+ 1615us) server_negotiation.cc:299] Negotiation successful
0810 11:41:03.409686 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":759,"thread_start_us":572,"threads_started":1}
I20260810 11:41:03.410307 26857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.382011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50497 (local address 127.3.240.195:43739)
0810 11:41:03.382769 (+ 758us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:03.382781 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:03.382816 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:41:03.384827 (+ 2011us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:03.384836 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:03.385249 (+ 413us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:03.385615 (+ 366us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:03.389482 (+ 3867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.391531 (+ 2049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.394565 (+ 3034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.395597 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.395792 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:03.401614 (+ 5822us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:03.401621 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:03.401632 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:03.401675 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:03.404736 (+ 3061us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:03.405700 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:03.405709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:03.405718 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:03.405956 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:03.407046 (+ 1090us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:03.407051 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:03.407054 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:03.407656 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:03.407793 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:03.409108 (+ 1315us) server_negotiation.cc:299] Negotiation successful
0810 11:41:03.409686 (+ 578us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":609,"thread_start_us":471,"threads_started":1}
I20260810 11:41:03.410109 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.381623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43739 (local address 127.3.240.193:50497)
0810 11:41:03.382602 (+ 979us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:03.382620 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:03.382733 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:03.386061 (+ 3328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:03.386077 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:03.387660 (+ 1583us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:03.389214 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:03.389246 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.391836 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.391840 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:03.394376 (+ 2536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:03.394394 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:03.395972 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:03.395981 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:03.396484 (+ 503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:03.397533 (+ 1049us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:03.397574 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:03.401359 (+ 3785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:03.404930 (+ 3571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:03.404935 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:03.404939 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:03.405531 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:03.406080 (+ 549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:03.406084 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:03.406087 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:03.406367 (+ 280us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:03.407841 (+ 1474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:03.407846 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:03.408194 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 11:41:03.408655 (+ 461us) client_negotiation.cc:240] Negotiation successful
0810 11:41:03.409700 (+ 1045us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":859,"mutex_wait_us":922,"spinlock_wait_cycles":41984,"thread_start_us":680,"threads_started":1}
W20260810 11:41:03.413619 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
W20260810 11:41:03.414052 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:41:03.414193 26301 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a707908d645c4e759a561ea752a0d844; no voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337
I20260810 11:41:03.414678 26851 raft_consensus.cc:2726] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:41:03.616760 26855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.616293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:32875)
0810 11:41:03.616488 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:03.616619 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 11:41:03.652324 26858 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:03.651141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:58063)
0810 11:41:03.651804 (+ 663us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:03.651986 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":322,"threads_started":1}
I20260810 11:41:04.131024 26859 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:04.130152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:58905)
0810 11:41:04.130692 (+ 540us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:04.130788 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":427,"thread_start_us":326,"threads_started":1}
I20260810 11:41:04.618777 26860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:04.617867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:41749)
0810 11:41:04.618506 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:04.618588 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":348,"threads_started":1}
I20260810 11:41:04.654466 26861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:04.653480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:34809)
0810 11:41:04.654164 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:04.654270 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":353,"threads_started":1}
I20260810 11:41:05.017042 26863 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:05.017238 26863 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:05.018062 26863 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
W20260810 11:41:05.019176 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
W20260810 11:41:05.019374 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:41:05.019459 26301 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a707908d645c4e759a561ea752a0d844; no voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337
I20260810 11:41:05.019763 26863 raft_consensus.cc:2726] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:41:05.133450 26864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:05.132362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:45209)
0810 11:41:05.133100 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:05.133213 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":392,"threads_started":1}
I20260810 11:41:05.620415 26865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:05.619685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:41179)
0810 11:41:05.620152 (+ 467us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:05.620235 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20260810 11:41:05.656739 26866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:05.655802 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:38719)
0810 11:41:05.656426 (+ 624us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:05.656529 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":344,"threads_started":1}
I20260810 11:41:06.135919 26867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:06.134623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:50975)
0810 11:41:06.135535 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:06.135679 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":464,"threads_started":1}
I20260810 11:41:06.622598 26868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:06.621716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:33807)
0810 11:41:06.622268 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:06.622357 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":269,"threads_started":1}
I20260810 11:41:06.659076 26870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:06.658138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:52209)
0810 11:41:06.658715 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:06.658807 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":325,"threads_started":1}
I20260810 11:41:07.138266 26871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:07.137439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:36335)
0810 11:41:07.137998 (+ 559us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:07.138089 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":299,"threads_started":1}
I20260810 11:41:07.574991 26872 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:07.575286 26872 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:07.576110 26872 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
W20260810 11:41:07.577097 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
W20260810 11:41:07.577292 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:41:07.577392 26301 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a707908d645c4e759a561ea752a0d844; no voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337
I20260810 11:41:07.577625 26872 raft_consensus.cc:2726] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:41:07.624403 26873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:07.623558 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:38143)
0810 11:41:07.624088 (+ 530us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:07.624200 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":424,"thread_start_us":281,"threads_started":1}
I20260810 11:41:07.660885 26874 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:07.660167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:36857)
0810 11:41:07.660626 (+ 459us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:07.660712 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":251,"threads_started":1}
I20260810 11:41:08.140183 26875 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:08.139302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:41485)
0810 11:41:08.139787 (+ 485us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:08.139878 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":258,"threads_started":1}
I20260810 11:41:08.626487 26877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:08.625723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:53709)
0810 11:41:08.626201 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:08.626295 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":265,"threads_started":1}
I20260810 11:41:08.663256 26878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:08.661937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:57189)
0810 11:41:08.662688 (+ 751us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:08.663055 (+ 367us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":422,"threads_started":1}
I20260810 11:41:09.142755 26880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:09.141427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:42231)
0810 11:41:09.142276 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:09.142415 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":608,"thread_start_us":361,"threads_started":1}
I20260810 11:41:09.628351 26881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:09.627582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:51441)
0810 11:41:09.628098 (+ 516us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:09.628181 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":422,"thread_start_us":300,"threads_started":1}
I20260810 11:41:09.665246 26882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:09.664395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:60475)
0810 11:41:09.664960 (+ 565us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:09.665053 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":252,"threads_started":1}
I20260810 11:41:10.144797 26883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:10.143991 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:53099)
0810 11:41:10.144505 (+ 514us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:10.144595 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":408,"thread_start_us":264,"threads_started":1}
I20260810 11:41:10.364316 26884 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:10.364562 26884 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:10.365224 26884 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
W20260810 11:41:10.366534 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
W20260810 11:41:10.366762 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:41:10.366861 26301 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a707908d645c4e759a561ea752a0d844; no voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337
I20260810 11:41:10.367305 26884 raft_consensus.cc:2726] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:41:10.630977 26885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:10.629749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:47595)
0810 11:41:10.630464 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:10.630733 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":353,"threads_started":1}
I20260810 11:41:10.667155 26886 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:10.666318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:51923)
0810 11:41:10.666783 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:10.666881 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":248,"threads_started":1}
I20260810 11:41:11.147392 26887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:11.146136 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:38329)
0810 11:41:11.146868 (+ 732us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:11.147125 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":355,"threads_started":1}
I20260810 11:41:11.633369 26888 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:11.632268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:48175)
0810 11:41:11.632952 (+ 684us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:11.633095 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":360,"threads_started":1}
I20260810 11:41:11.669445 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:11.668491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:52791)
0810 11:41:11.669130 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:11.669234 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":309,"threads_started":1}
I20260810 11:41:12.149870 26890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:12.148874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:56699)
0810 11:41:12.149519 (+ 645us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:12.149626 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":409,"threads_started":1}
I20260810 11:41:12.635663 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:12.634620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:43009)
0810 11:41:12.635357 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:12.635473 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":450,"threads_started":1}
I20260810 11:41:12.671873 26893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:12.670821 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:32997)
0810 11:41:12.671559 (+ 738us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:12.671666 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":612,"thread_start_us":472,"threads_started":1}
I20260810 11:41:13.152439 26894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:13.151242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:53181)
0810 11:41:13.152072 (+ 830us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:13.152215 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":695,"thread_start_us":525,"threads_started":1}
I20260810 11:41:13.637759 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:13.636946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:41203)
0810 11:41:13.637479 (+ 533us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:13.637569 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":291,"threads_started":1}
I20260810 11:41:13.674414 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:13.673260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:54847)
0810 11:41:13.673986 (+ 726us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:13.674188 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":385,"threads_started":1}
I20260810 11:41:14.155277 26899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:14.154122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:57779)
0810 11:41:14.154800 (+ 678us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:14.154911 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":381,"threads_started":1}
I20260810 11:41:14.639729 26900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:14.638875 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:50069)
0810 11:41:14.639450 (+ 575us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:14.639548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":357,"threads_started":1}
I20260810 11:41:14.676879 26901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:14.675925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:33447)
0810 11:41:14.676551 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:14.676657 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":330,"threads_started":1}
I20260810 11:41:14.906055 26902 raft_consensus.cc:482] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:14.906316 26902 raft_consensus.cc:504] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a707908d645c4e759a561ea752a0d844" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38057 } } peers { permanent_uuid: "b035d45e24344551a34c1b0529a03179" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 41943 } } peers { permanent_uuid: "e3f5a39dbc264c64a395c3e18dedc337" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43739 } }
I20260810 11:41:14.907428 26902 leader_election.cc:288] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943), e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739)
W20260810 11:41:14.908668 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer b035d45e24344551a34c1b0529a03179 (127.3.240.194:41943): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
W20260810 11:41:14.908944 26301 leader_election.cc:341] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e3f5a39dbc264c64a395c3e18dedc337 (127.3.240.195:43739): Not found: Tablet not found: cdc424bb5cc44d77aaa7db72acb6cbc4
I20260810 11:41:14.909083 26301 leader_election.cc:302] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a707908d645c4e759a561ea752a0d844; no voters: b035d45e24344551a34c1b0529a03179, e3f5a39dbc264c64a395c3e18dedc337
I20260810 11:41:14.909437 26902 raft_consensus.cc:2726] T cdc424bb5cc44d77aaa7db72acb6cbc4 P a707908d645c4e759a561ea752a0d844 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260810 11:41:15.157588 26903 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:15.156723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:51917)
0810 11:41:15.157269 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:15.157392 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":287,"threads_started":1}
I20260810 11:41:15.641587 26904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:15.640758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:54709)
0810 11:41:15.641304 (+ 546us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:15.641406 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":324,"threads_started":1}
I20260810 11:41:15.679122 26906 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:15.677947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:53637)
0810 11:41:15.678708 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:15.678878 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":354,"threads_started":1}
I20260810 11:41:16.159507 26907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:16.158583 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:50655)
0810 11:41:16.159155 (+ 572us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:16.159251 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":473,"thread_start_us":282,"threads_started":1}
I20260810 11:41:16.643733 26908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:16.642806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:41201)
0810 11:41:16.643358 (+ 552us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:16.643483 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":336,"threads_started":1}
I20260810 11:41:16.680810 26911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:16.680024 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:38327)
0810 11:41:16.680520 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:16.680604 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":386,"thread_start_us":276,"threads_started":1}
I20260810 11:41:17.161708 26913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.160775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.194:47735)
0810 11:41:17.161413 (+ 638us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:17.161516 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":323,"threads_started":1}
I20260810 11:41:17.645571 26916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.644795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.193:37927)
0810 11:41:17.645306 (+ 511us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:17.645401 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":264,"threads_started":1}
I20260810 11:41:17.661374 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 26234
I20260810 11:41:17.683034 26919 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.682162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46745 (local address 127.3.240.195:54781)
0810 11:41:17.682660 (+ 498us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:17.682790 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:46745: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":273,"threads_started":1}
I20260810 11:41:17.686096 26918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.662657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.3.240.193:38057)
0810 11:41:17.663711 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:17.663727 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:17.663796 (+ 69us) server_negotiation.cc:407] Connection header received
0810 11:41:17.663916 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:17.663930 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:17.664384 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:17.664667 (+ 283us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:17.667928 (+ 3261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.669269 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.671266 (+ 1997us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.672041 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.672194 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:17.679321 (+ 7127us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:17.679337 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:17.679355 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:17.679404 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:17.682829 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.683681 (+ 852us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.683688 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.683691 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.683839 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.684275 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.684279 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.684281 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.684780 (+ 499us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:17.685021 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:17.685402 (+ 381us) server_negotiation.cc:299] Negotiation successful
0810 11:41:17.685781 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":356,"threads_started":1}
I20260810 11:41:17.687552 26465 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:17.787061 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 26234
I20260810 11:41:17.824028 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 26475
I20260810 11:41:17.844596 26921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.825187 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45174 (local address 127.3.240.194:41943)
0810 11:41:17.826101 (+ 914us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:17.826107 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:17.826128 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:41:17.826206 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:17.826213 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:17.826311 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:17.826447 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:17.828559 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.829651 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.831991 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.832840 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.833024 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:17.837249 (+ 4225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:17.837255 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:17.837258 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:17.837293 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:17.841557 (+ 4264us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.842335 (+ 778us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.842340 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.842344 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.842487 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.842886 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.842899 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.842902 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.843238 (+ 336us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:17.843533 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:17.843847 (+ 314us) server_negotiation.cc:299] Negotiation successful
0810 11:41:17.844085 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":777,"thread_start_us":688,"threads_started":1}
I20260810 11:41:17.845288 26652 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:17.936590 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 26475
I20260810 11:41:17.970640 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 26663
I20260810 11:41:17.990252 26924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:17.971771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46678 (local address 127.3.240.195:43739)
0810 11:41:17.972675 (+ 904us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:17.972679 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:17.972705 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:41:17.972762 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:17.972766 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:17.972842 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:17.972937 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:17.976761 (+ 3824us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.977553 (+ 792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.979066 (+ 1513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:17.979577 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:17.979706 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:17.982726 (+ 3020us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:17.982735 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:17.982739 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:17.982795 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:17.986163 (+ 3368us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.987946 (+ 1783us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.987951 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.987954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.988176 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:17.988851 (+ 675us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:17.988855 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:17.988858 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:17.989161 (+ 303us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:17.989418 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:17.989748 (+ 330us) server_negotiation.cc:299] Negotiation successful
0810 11:41:17.989956 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":789,"thread_start_us":613,"threads_started":1}
I20260810 11:41:17.991297 26838 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:18.084805 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 26663
2026-08-10T11:41:18Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26128 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:41:18 dist-test-slave-drl0 krb5kdc[26929](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:41:18 dist-test-slave-drl0 krb5kdc[26929](info): set up 2 sockets
Aug 10 11:41:18 dist-test-slave-drl0 krb5kdc[26929](info): commencing operation
krb5kdc: starting...
W20260810 11:41:19.465055 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.001s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:41:19 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:41:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:41:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:41:19.798724 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:41:20.185668 26949 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:20.186106 26949 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:20.186283 26949 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:20.195552 26949 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 11:41:20.195653 26949 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 11:41:20.195751 26949 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 11:41:20.195835 26949 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:41:20.195943 26949 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:20.196051 26949 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:20.196136 26949 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:41:20.196221 26949 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:41:20.206982 26949 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:20.209146 26949 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:20.210968 26949 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:20.227452 26956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:20.227478 26957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:20.227424 26959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:20.228205 26949 server_base.cc:1034] running on GCE node
I20260810 11:41:20.228937 26949 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:20.230857 26949 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:20.232290 26949 hybrid_clock.cc:648] HybridClock initialized: now 1786362080232186 us; error 70 us; skew 500 ppm
I20260810 11:41:20.232738 26949 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:20 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362080, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:20.236683 26949 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:41:20.238344 26949 webserver.cc:460] Webserver started at http://127.3.240.254:39539/ using document root <none> and password file <none>
I20260810 11:41:20.239147 26949 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:20.239305 26949 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:20.239639 26949 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:41:20.242681 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "077fb73607ea40f2889e37faedeb82c3"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "f894f34434b2f653b6db653c6359e8e3"
server_key_iv: "8c53fe6359f3e078cdcc4a0aaf3fab78"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.243557 26949 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "077fb73607ea40f2889e37faedeb82c3"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "f894f34434b2f653b6db653c6359e8e3"
server_key_iv: "8c53fe6359f3e078cdcc4a0aaf3fab78"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.250305 26949 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 11:41:20.255231 26974 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:20.256955 26949 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:41:20.257230 26949 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "077fb73607ea40f2889e37faedeb82c3"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "f894f34434b2f653b6db653c6359e8e3"
server_key_iv: "8c53fe6359f3e078cdcc4a0aaf3fab78"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.257524 26949 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:20.274698 26949 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:20.280228 26949 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:20.280570 26949 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:20.303531 26949 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:36325
I20260810 11:41:20.303543 27037 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:36325 every 8 connection(s)
I20260810 11:41:20.304903 26949 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:41:20.308758 27038 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:20.314999 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 26949
I20260810 11:41:20.315245 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:41:20.316139 4035 external_mini_cluster.cc:1428] Setting key d2bed96e1e98dc799cf14f164973c2c9
I20260810 11:41:20.320811 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap starting.
I20260810 11:41:20.324030 27038 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:20.325207 27038 log.cc:824] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:20.328155 27038 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: No bootstrap required, opened a new log
Aug 10 11:41:20 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:20.337186 27038 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:20.337618 27038 raft_consensus.cc:374] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:20.337742 27038 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Initialized, Role: FOLLOWER
I20260810 11:41:20.338490 27038 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:20.338798 27038 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:41:20.338984 27038 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:41:20.339242 27038 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:20.342530 27041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:20.318497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60182 (local address 127.3.240.254:36325)
0810 11:41:20.319445 (+ 948us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:20.319457 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:20.319499 (+ 42us) server_negotiation.cc:407] Connection header received
0810 11:41:20.321247 (+ 1748us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:20.321269 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:20.321648 (+ 379us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:20.322134 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:20.323774 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:20.325168 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:20.326968 (+ 1800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:20.327654 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:20.327861 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:20.333327 (+ 5466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:20.333334 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:20.333350 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:20.333397 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:20.337411 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:20.338417 (+ 1006us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:20.338424 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:20.338430 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:20.338643 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:20.339123 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:20.339128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:20.339131 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:20.339897 (+ 766us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:20.340240 (+ 343us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:20.340724 (+ 484us) server_negotiation.cc:299] Negotiation successful
0810 11:41:20.341198 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":729,"thread_start_us":492,"threads_started":1}
I20260810 11:41:20.342675 27038 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:20.343308 27038 leader_election.cc:302] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077fb73607ea40f2889e37faedeb82c3; no voters:
I20260810 11:41:20.344013 27038 leader_election.cc:288] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:41:20.344133 27043 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:41:20.346062 27038 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:41:20.346297 27043 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 LEADER]: Becoming Leader. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Running, Role: LEADER
I20260810 11:41:20.346872 27043 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:20.350174 27044 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:41:20.350157 27045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077fb73607ea40f2889e37faedeb82c3. Latest consensus state: current_term: 1 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:41:20.350634 27045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:20.350632 27044 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:20.351686 27054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:41:20.360618 27054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:41:20.370460 27054 catalog_manager.cc:1347] Generated new cluster ID: 3c0e3a89c88f451dae2c03058982b736
I20260810 11:41:20.370771 27054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:41:20.388597 27054 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:41:20.390396 27054 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:41:20.403092 27054 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Generated new TSK 0
I20260810 11:41:20.403954 27054 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:41:20.494369 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:41:20.882179 27072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:20.882469 27072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:20.882630 27072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:20.890547 27072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:20.890664 27072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:20.890782 27072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:41:20.902000 27072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:20.903535 27072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:20.904884 27072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:20.919199 27079 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:20.919345 27086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:20.919382 27082 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:20.920637 27072 server_base.cc:1034] running on GCE node
I20260810 11:41:20.921284 27072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:20.922719 27072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:20.924073 27072 hybrid_clock.cc:648] HybridClock initialized: now 1786362080923986 us; error 70 us; skew 500 ppm
I20260810 11:41:20.924455 27072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:20 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362080, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:20.927932 27072 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:41:20.934720 27072 webserver.cc:460] Webserver started at http://127.3.240.193:39325/ using document root <none> and password file <none>
I20260810 11:41:20.935796 27072 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:20.936003 27072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:20.936376 27072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:41:20.940297 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "7292538215da4edba214fd1d1ce8e69d"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "ebf5f44ef71179a26de8c28bf1419c20"
server_key_iv: "7acab3ffb18b1ad9b02a94f4f6b063c2"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.941473 27072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "7292538215da4edba214fd1d1ce8e69d"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "ebf5f44ef71179a26de8c28bf1419c20"
server_key_iv: "7acab3ffb18b1ad9b02a94f4f6b063c2"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.947340 27072 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 11:41:20.951406 27098 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:20.952965 27072 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 11:41:20.953228 27072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "7292538215da4edba214fd1d1ce8e69d"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "ebf5f44ef71179a26de8c28bf1419c20"
server_key_iv: "7acab3ffb18b1ad9b02a94f4f6b063c2"
server_key_version: "encryptionkey@0"
I20260810 11:41:20.953465 27072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:20.971208 27072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:20.975716 27072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:20.976020 27072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:20.977164 27072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:20.978638 27072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:20.978745 27072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:20.978904 27072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:20.978996 27072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260810 11:41:21.020522 27072 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33489
I20260810 11:41:21.020648 27251 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33489 every 8 connection(s)
I20260810 11:41:21.022147 27072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:41:21.029320 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 27072
I20260810 11:41:21.029608 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:41:21.030498 4035 external_mini_cluster.cc:1428] Setting key c1dfde64dd3b538847c2e8a1db6bb60a
Aug 10 11:41:21 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362080, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:21.045876 27257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:21.025665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:37975 (local address 127.3.240.254:36325)
0810 11:41:21.026209 (+ 544us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:21.026216 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:21.027664 (+ 1448us) server_negotiation.cc:407] Connection header received
0810 11:41:21.029848 (+ 2184us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:21.029853 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:21.029950 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:21.030080 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:21.031854 (+ 1774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.032733 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.034250 (+ 1517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.034702 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.034834 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:21.039454 (+ 4620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:21.039459 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:21.039462 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:21.039496 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:21.042612 (+ 3116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:21.043477 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:21.043482 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:21.043488 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:21.043616 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:21.044078 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:21.044083 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:21.044088 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:21.044375 (+ 287us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:21.044539 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:21.045269 (+ 730us) server_negotiation.cc:299] Negotiation successful
0810 11:41:21.045495 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":397,"thread_start_us":275,"threads_started":1}
I20260810 11:41:21.047070 27258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:21.026169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:37975)
0810 11:41:21.027409 (+ 1240us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:21.027475 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:21.029463 (+ 1988us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:21.030337 (+ 874us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:21.030345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:21.030972 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:21.031674 (+ 702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:21.031702 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.032897 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.032913 (+ 16us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:21.034107 (+ 1194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:21.034119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.034983 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.034994 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:21.035347 (+ 353us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:21.036243 (+ 896us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:21.036277 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:21.039205 (+ 2928us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:21.042786 (+ 3581us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:21.042795 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:21.042811 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:21.043344 (+ 533us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:21.043737 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:21.043742 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:21.043754 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:21.043951 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:21.044515 (+ 564us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:21.044523 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:21.044871 (+ 348us) client_negotiation.cc:769] Sending connection context
0810 11:41:21.045144 (+ 273us) client_negotiation.cc:240] Negotiation successful
0810 11:41:21.045671 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":457,"threads_started":1}
I20260810 11:41:21.048986 27252 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:21.049327 27252 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:21.050163 27252 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:21.052757 26999 ts_manager.cc:194] Registered new tserver with Master: 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:21.057085 26999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:37975
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:41:21.111994 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:41:21.479739 27266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:21.480028 27266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:21.480234 27266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:21.488436 27266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:21.488549 27266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:21.488716 27266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:41:21.498724 27266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:21.500114 27266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:21.501674 27266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:21.515739 27273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:21.515869 27278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:21.515739 27274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:21.517563 27266 server_base.cc:1034] running on GCE node
I20260810 11:41:21.518263 27266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:21.519359 27266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:21.520629 27266 hybrid_clock.cc:648] HybridClock initialized: now 1786362081520576 us; error 37 us; skew 500 ppm
I20260810 11:41:21.521000 27266 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:21 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:21.524648 27266 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:41:21.526098 27266 webserver.cc:460] Webserver started at http://127.3.240.194:33159/ using document root <none> and password file <none>
I20260810 11:41:21.526754 27266 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:21.526872 27266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:21.527212 27266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:41:21.530282 27266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "90b53b7fbc83441ea6b542a1a39dd792"
format_stamp: "Formatted at 2026-08-10 11:41:21 on dist-test-slave-drl0"
server_key: "048375057b0a55e72c80f798f02e268e"
server_key_iv: "223c42e7f241c04f4219299f02048453"
server_key_version: "encryptionkey@0"
I20260810 11:41:21.531095 27266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "90b53b7fbc83441ea6b542a1a39dd792"
format_stamp: "Formatted at 2026-08-10 11:41:21 on dist-test-slave-drl0"
server_key: "048375057b0a55e72c80f798f02e268e"
server_key_iv: "223c42e7f241c04f4219299f02048453"
server_key_version: "encryptionkey@0"
I20260810 11:41:21.536401 27266 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 11:41:21.540408 27287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:21.542423 27266 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:41:21.542773 27266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "90b53b7fbc83441ea6b542a1a39dd792"
format_stamp: "Formatted at 2026-08-10 11:41:21 on dist-test-slave-drl0"
server_key: "048375057b0a55e72c80f798f02e268e"
server_key_iv: "223c42e7f241c04f4219299f02048453"
server_key_version: "encryptionkey@0"
I20260810 11:41:21.543217 27266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:21.567126 27266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:21.572908 27266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:21.573312 27266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:21.574394 27266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:21.576503 27266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:21.576620 27266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:21.576789 27266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:21.576867 27266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:21.619545 27266 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38281
I20260810 11:41:21.619588 27440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38281 every 8 connection(s)
I20260810 11:41:21.621155 27266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:41:21.624610 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 27266
I20260810 11:41:21.624845 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:41:21.625370 4035 external_mini_cluster.cc:1428] Setting key 2ea95f2f51207fcd06aaddb2da040ca4
Aug 10 11:41:21 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362081, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:21.643567 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:21.624529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42653 (local address 127.3.240.254:36325)
0810 11:41:21.625056 (+ 527us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:21.625062 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:21.626088 (+ 1026us) server_negotiation.cc:407] Connection header received
0810 11:41:21.628614 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:21.628619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:21.628713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:21.628832 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:21.630340 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.631135 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.632683 (+ 1548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.633129 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.633281 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:21.636920 (+ 3639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:21.636925 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:21.636928 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:21.636968 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:21.639764 (+ 2796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:21.640765 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:21.640772 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:21.640775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:21.641016 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:21.641650 (+ 634us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:21.641655 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:21.641657 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:21.642000 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:21.642209 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:21.642916 (+ 707us) server_negotiation.cc:299] Negotiation successful
0810 11:41:21.643184 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":252,"threads_started":1}
I20260810 11:41:21.644553 27444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:21.625050 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:42653)
0810 11:41:21.625882 (+ 832us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:21.625945 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:21.628324 (+ 2379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:21.629077 (+ 753us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:21.629085 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:21.629610 (+ 525us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:21.630164 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:21.630182 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.631298 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.631302 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:21.632522 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:21.632535 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:21.633369 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:21.633376 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:21.633610 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:21.634353 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:21.634383 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:21.636677 (+ 2294us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:21.639933 (+ 3256us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:21.639942 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:21.639959 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:21.640496 (+ 537us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:21.641187 (+ 691us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:21.641195 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:21.641200 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:21.641472 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:21.642182 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:21.642191 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:21.642532 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 11:41:21.642823 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 11:41:21.643399 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":353,"threads_started":1}
I20260810 11:41:21.646250 27441 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:21.646641 27441 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:21.647470 27441 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:21.649251 26999 ts_manager.cc:194] Registered new tserver with Master: 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281)
I20260810 11:41:21.650164 26999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:42653
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:41:21.703763 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:41:22.061417 27252 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
W20260810 11:41:22.074996 27448 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:22.075436 27448 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:22.075652 27448 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:22.084400 27448 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:22.084618 27448 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:22.084836 27448 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:41:22.097055 27448 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:22.098781 27448 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:22.100252 27448 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:22.113947 27457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:22.113857 27458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:22.113909 27461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:22.114414 27448 server_base.cc:1034] running on GCE node
I20260810 11:41:22.115245 27448 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:22.116204 27448 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:22.117465 27448 hybrid_clock.cc:648] HybridClock initialized: now 1786362082117402 us; error 48 us; skew 500 ppm
I20260810 11:41:22.117810 27448 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:22 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:22.121289 27448 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:41:22.122761 27448 webserver.cc:460] Webserver started at http://127.3.240.195:37017/ using document root <none> and password file <none>
I20260810 11:41:22.123521 27448 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:22.123670 27448 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:22.123992 27448 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:41:22.126863 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "dc59dc6710f64e1685fba3be4975b94b"
format_stamp: "Formatted at 2026-08-10 11:41:22 on dist-test-slave-drl0"
server_key: "3681706a76b737c455716b81f8e64c1e"
server_key_iv: "6269da1cc0bd4bea1595f09a95bfa1d4"
server_key_version: "encryptionkey@0"
I20260810 11:41:22.127727 27448 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "dc59dc6710f64e1685fba3be4975b94b"
format_stamp: "Formatted at 2026-08-10 11:41:22 on dist-test-slave-drl0"
server_key: "3681706a76b737c455716b81f8e64c1e"
server_key_iv: "6269da1cc0bd4bea1595f09a95bfa1d4"
server_key_version: "encryptionkey@0"
I20260810 11:41:22.133615 27448 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:41:22.137193 27472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:22.139015 27448 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:41:22.139217 27448 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "dc59dc6710f64e1685fba3be4975b94b"
format_stamp: "Formatted at 2026-08-10 11:41:22 on dist-test-slave-drl0"
server_key: "3681706a76b737c455716b81f8e64c1e"
server_key_iv: "6269da1cc0bd4bea1595f09a95bfa1d4"
server_key_version: "encryptionkey@0"
I20260810 11:41:22.139860 27448 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:22.172900 27448 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:22.177409 27448 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:22.177721 27448 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:22.178788 27448 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:22.180684 27448 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:22.180815 27448 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:22.181293 27448 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:22.181450 27448 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:22.221508 27448 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40767
I20260810 11:41:22.221527 27632 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40767 every 8 connection(s)
I20260810 11:41:22.223488 27448 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:41:22.230281 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 27448
I20260810 11:41:22.230515 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:41:22.231088 4035 external_mini_cluster.cc:1428] Setting key 1cab5a405c9d1dee7f5b41abd2cc6634
Aug 10 11:41:22 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362082, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:22.246004 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.227067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:45175 (local address 127.3.240.254:36325)
0810 11:41:22.227773 (+ 706us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.227782 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.228733 (+ 951us) server_negotiation.cc:407] Connection header received
0810 11:41:22.230736 (+ 2003us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.230742 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.230850 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.231004 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:22.232770 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.233711 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.235400 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.235803 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.235945 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.239791 (+ 3846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:22.239796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:22.239800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:22.239832 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:22.242720 (+ 2888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.243531 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.243537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.243539 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.243662 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.244118 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.244123 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.244125 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.244390 (+ 265us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:22.244559 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.245382 (+ 823us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.245642 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":281,"threads_started":1}
I20260810 11:41:22.247382 27637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.227513 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:45175)
0810 11:41:22.228540 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.228596 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.230416 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.231252 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.231263 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.231826 (+ 563us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:22.232609 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.232629 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.233880 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.233885 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.235260 (+ 1375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.235272 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.236045 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.236053 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.236282 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.236994 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:22.237016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:22.239564 (+ 2548us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:22.242891 (+ 3327us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:22.242900 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:22.242916 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:22.243389 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:22.243779 (+ 390us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:22.243784 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:22.243788 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:22.243985 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:22.244523 (+ 538us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:22.244531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:22.244880 (+ 349us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.245202 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.245771 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":349,"threads_started":1}
I20260810 11:41:22.249009 27633 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:22.249346 27633 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:22.250228 27633 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:22.251839 26999 ts_manager.cc:194] Registered new tserver with Master: dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767)
I20260810 11:41:22.252678 26999 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:45175
I20260810 11:41:22.259924 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:41:22.279472 27636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.265042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60186 (local address 127.3.240.254:36325)
0810 11:41:22.265260 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.265264 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.265827 (+ 563us) server_negotiation.cc:407] Connection header received
0810 11:41:22.265936 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.265940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.266059 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.266196 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:22.267719 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.268478 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.270012 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.270424 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.270535 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.273435 (+ 2900us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:22.273440 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:22.273443 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:22.273473 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:22.276218 (+ 2745us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.276938 (+ 720us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.276942 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.276944 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.277067 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.277753 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.277760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.277762 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.278230 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:22.278494 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.279021 (+ 527us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.279224 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 11:41:22.284490 26999 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:41:22.291961 26999 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:41:22.318351 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.305156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40767 (local address 127.0.0.1:42750)
0810 11:41:22.305918 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.305963 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.306120 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.307413 (+ 1293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.307425 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.307470 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:22.307944 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.307961 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.310642 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.310650 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.313914 (+ 3264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.313937 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.316952 (+ 3015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.316961 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.317278 (+ 317us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.317362 (+ 84us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.317530 (+ 168us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.317810 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":779,"threads_started":1}
I20260810 11:41:22.320508 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.305629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42750 (local address 127.3.240.195:40767)
0810 11:41:22.306358 (+ 729us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.306367 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.306401 (+ 34us) server_negotiation.cc:407] Connection header received
0810 11:41:22.306497 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.306504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.306748 (+ 244us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.306999 (+ 251us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:22.308238 (+ 1239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.310426 (+ 2188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.314173 (+ 3747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.316254 (+ 2081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.316443 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.316754 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.319185 (+ 2431us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.319435 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":1556,"threads_started":1}
I20260810 11:41:22.328893 27548 tablet_service.cc:1467] Processing CreateTablet for tablet 42556d8e9227411299fc5e859d7ffe0d (DEFAULT_TABLE table=test-table [id=3a6b7faf73234ef582f796869892b621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:22.330595 27548 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42556d8e9227411299fc5e859d7ffe0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:22.331780 27649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.322803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:40788)
0810 11:41:22.323034 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.323052 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.323157 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.324027 (+ 870us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.324032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.324055 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:22.324264 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.324274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.326617 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.326625 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.329071 (+ 2446us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.329088 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.330977 (+ 1889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.330984 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.331174 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.331217 (+ 43us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.331316 (+ 99us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.331492 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":69,"mutex_wait_us":5}
I20260810 11:41:22.332581 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.321912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40788 (local address 127.3.240.193:33489)
0810 11:41:22.322769 (+ 857us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.322778 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.323271 (+ 493us) server_negotiation.cc:407] Connection header received
0810 11:41:22.323372 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.323379 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.323620 (+ 241us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.323806 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:22.324415 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.326328 (+ 1913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.329293 (+ 2965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.330772 (+ 1479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.330920 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.331183 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.332073 (+ 890us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.332258 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":659,"thread_start_us":439,"threads_started":1}
I20260810 11:41:22.333191 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.307026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39794 (local address 127.3.240.194:38281)
0810 11:41:22.321949 (+ 14923us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.321961 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.322006 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:41:22.322175 (+ 169us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.322187 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.322723 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.323020 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:22.324026 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.325671 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.330127 (+ 4456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.331947 (+ 1820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.332098 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.332290 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.332686 (+ 396us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.332875 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14658,"thread_start_us":14337,"threads_started":1}
I20260810 11:41:22.333456 27652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.306728 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38281 (local address 127.0.0.1:39794)
0810 11:41:22.320672 (+ 13944us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.320697 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.320840 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.323263 (+ 2423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.323269 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.323295 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:22.323799 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.323815 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.325883 (+ 2068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.325888 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.329923 (+ 4035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.329938 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.332259 (+ 2321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.332265 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.332462 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.332493 (+ 31us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.332585 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.332965 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13782,"thread_start_us":14238,"threads_started":1}
I20260810 11:41:22.336402 27165 tablet_service.cc:1467] Processing CreateTablet for tablet 42556d8e9227411299fc5e859d7ffe0d (DEFAULT_TABLE table=test-table [id=3a6b7faf73234ef582f796869892b621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:22.336977 27348 tablet_service.cc:1467] Processing CreateTablet for tablet 42556d8e9227411299fc5e859d7ffe0d (DEFAULT_TABLE table=test-table [id=3a6b7faf73234ef582f796869892b621]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:22.337585 27165 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42556d8e9227411299fc5e859d7ffe0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:22.338248 27348 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 42556d8e9227411299fc5e859d7ffe0d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:22.347013 27658 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Bootstrap starting.
I20260810 11:41:22.350493 27660 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Bootstrap starting.
I20260810 11:41:22.350912 27658 tablet_bootstrap.cc:654] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:22.352332 27658 log.cc:824] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:22.353389 27662 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Bootstrap starting.
I20260810 11:41:22.354318 27660 tablet_bootstrap.cc:654] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:22.355566 27660 log.cc:824] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:22.356802 27658 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: No bootstrap required, opened a new log
I20260810 11:41:22.357115 27658 ts_tablet_manager.cc:1397] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Time spent bootstrapping tablet: real 0.012s user 0.001s sys 0.007s
I20260810 11:41:22.357468 27662 tablet_bootstrap.cc:654] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:22.358587 27662 log.cc:824] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:22.358639 27660 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: No bootstrap required, opened a new log
I20260810 11:41:22.358896 27660 ts_tablet_manager.cc:1397] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 11:41:22.361140 27662 tablet_bootstrap.cc:492] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: No bootstrap required, opened a new log
I20260810 11:41:22.361421 27662 ts_tablet_manager.cc:1397] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.002s
I20260810 11:41:22.368916 27662 raft_consensus.cc:348] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.369292 27662 raft_consensus.cc:374] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:22.369024 27658 raft_consensus.cc:348] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.369396 27662 raft_consensus.cc:729] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b53b7fbc83441ea6b542a1a39dd792, State: Initialized, Role: FOLLOWER
I20260810 11:41:22.369446 27658 raft_consensus.cc:374] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:22.369541 27658 raft_consensus.cc:729] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc59dc6710f64e1685fba3be4975b94b, State: Initialized, Role: FOLLOWER
I20260810 11:41:22.369699 27660 raft_consensus.cc:348] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.370101 27660 raft_consensus.cc:374] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:22.370208 27660 raft_consensus.cc:729] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7292538215da4edba214fd1d1ce8e69d, State: Initialized, Role: FOLLOWER
I20260810 11:41:22.370229 27662 consensus_queue.cc:260] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.370178 27658 consensus_queue.cc:260] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.371726 27441 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
I20260810 11:41:22.372215 27662 ts_tablet_manager.cc:1428] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Time spent starting tablet: real 0.011s user 0.009s sys 0.002s
I20260810 11:41:22.371160 27660 consensus_queue.cc:260] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
W20260810 11:41:22.374106 27442 tablet.cc:2369] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:41:22.376428 27660 ts_tablet_manager.cc:1428] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Time spent starting tablet: real 0.017s user 0.009s sys 0.006s
I20260810 11:41:22.377391 27633 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
I20260810 11:41:22.377990 27658 ts_tablet_manager.cc:1428] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Time spent starting tablet: real 0.021s user 0.018s sys 0.003s
W20260810 11:41:22.475653 27634 tablet.cc:2369] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:41:22.482760 27667 raft_consensus.cc:482] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:22.483126 27667 raft_consensus.cc:504] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.484880 27667 leader_election.cc:288] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767), 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:22.492383 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.485505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:54483 (local address 127.3.240.195:40767)
0810 11:41:22.485766 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.485773 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.485995 (+ 222us) server_negotiation.cc:407] Connection header received
0810 11:41:22.486255 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.486260 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.486358 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.486412 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:22.487298 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.488209 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.490166 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.491244 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.491395 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.491457 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.491946 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.492094 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260810 11:41:22.492401 27671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.485177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40767 (local address 127.3.240.194:54483)
0810 11:41:22.485873 (+ 696us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.485890 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.486063 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.486587 (+ 524us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.486590 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.486637 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:22.487123 (+ 486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.487140 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.488352 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.488357 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.489992 (+ 1635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.490005 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.491471 (+ 1466us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.491479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.491732 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.491781 (+ 49us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.491890 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.492037 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":344,"threads_started":1}
I20260810 11:41:22.493010 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.486081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:60089 (local address 127.3.240.193:33489)
0810 11:41:22.486285 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.486291 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.486658 (+ 367us) server_negotiation.cc:407] Connection header received
0810 11:41:22.486819 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.486824 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.486911 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.487048 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:22.487712 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.488539 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.490779 (+ 2240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.491975 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.492125 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.492207 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.492702 (+ 495us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.492836 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260810 11:41:22.493144 27672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.485899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.3.240.194:60089)
0810 11:41:22.486552 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:22.486566 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:22.486671 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:22.487063 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:22.487066 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:22.487086 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:22.487499 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.487510 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.488724 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.488728 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.490580 (+ 1852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:22.490596 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.492230 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.492237 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:22.492451 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.492495 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 11:41:22.492656 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 11:41:22.492800 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":364,"threads_started":1}
I20260810 11:41:22.493605 27574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42556d8e9227411299fc5e859d7ffe0d" candidate_uuid: "90b53b7fbc83441ea6b542a1a39dd792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc59dc6710f64e1685fba3be4975b94b" is_pre_election: true
I20260810 11:41:22.494225 27574 raft_consensus.cc:2454] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b53b7fbc83441ea6b542a1a39dd792 in term 0.
I20260810 11:41:22.494117 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42556d8e9227411299fc5e859d7ffe0d" candidate_uuid: "90b53b7fbc83441ea6b542a1a39dd792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7292538215da4edba214fd1d1ce8e69d" is_pre_election: true
I20260810 11:41:22.494673 27195 raft_consensus.cc:2454] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 90b53b7fbc83441ea6b542a1a39dd792 in term 0.
I20260810 11:41:22.495025 27288 leader_election.cc:302] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b53b7fbc83441ea6b542a1a39dd792, dc59dc6710f64e1685fba3be4975b94b; no voters:
I20260810 11:41:22.495460 27667 raft_consensus.cc:2781] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:41:22.495630 27667 raft_consensus.cc:482] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:41:22.495750 27667 raft_consensus.cc:3037] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:22.498225 27667 raft_consensus.cc:504] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.498806 27667 leader_election.cc:288] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [CANDIDATE]: Term 1 election: Requested vote from peers dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767), 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:22.499734 27574 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42556d8e9227411299fc5e859d7ffe0d" candidate_uuid: "90b53b7fbc83441ea6b542a1a39dd792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dc59dc6710f64e1685fba3be4975b94b"
I20260810 11:41:22.499734 27195 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "42556d8e9227411299fc5e859d7ffe0d" candidate_uuid: "90b53b7fbc83441ea6b542a1a39dd792" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7292538215da4edba214fd1d1ce8e69d"
I20260810 11:41:22.499966 27195 raft_consensus.cc:3037] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:22.499961 27574 raft_consensus.cc:3037] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:22.503407 27195 raft_consensus.cc:2454] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b53b7fbc83441ea6b542a1a39dd792 in term 1.
I20260810 11:41:22.503422 27574 raft_consensus.cc:2454] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 90b53b7fbc83441ea6b542a1a39dd792 in term 1.
I20260810 11:41:22.503837 27288 leader_election.cc:302] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7292538215da4edba214fd1d1ce8e69d, 90b53b7fbc83441ea6b542a1a39dd792; no voters:
I20260810 11:41:22.504155 27667 raft_consensus.cc:2781] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:41:22.504655 27667 raft_consensus.cc:686] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 LEADER]: Becoming Leader. State: Replica: 90b53b7fbc83441ea6b542a1a39dd792, State: Running, Role: LEADER
I20260810 11:41:22.505259 27667 consensus_queue.cc:237] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:22.510279 26998 catalog_manager.cc:5495] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 reported cstate change: term changed from 0 to 1, leader changed from <none> to 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "90b53b7fbc83441ea6b542a1a39dd792" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } health_report { overall_health: UNKNOWN } } }
W20260810 11:41:22.525084 27256 tablet.cc:2369] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:41:22.532572 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.526996 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39802 (local address 127.3.240.194:38281)
0810 11:41:22.527221 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.527227 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.527248 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:41:22.527503 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.527508 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.527595 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.527754 (+ 159us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:41:22.528167 (+ 413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.528909 (+ 742us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.530348 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.530776 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.530918 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.531428 (+ 510us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:41:22.531932 (+ 504us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:41:22.532092 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.532194 (+ 102us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.532274 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 11:41:22.541021 27195 raft_consensus.cc:1261] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Refusing update from remote peer 90b53b7fbc83441ea6b542a1a39dd792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:41:22.541025 27574 raft_consensus.cc:1261] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Refusing update from remote peer 90b53b7fbc83441ea6b542a1a39dd792: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:41:22.542443 27674 consensus_queue.cc:1035] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:41:22.543141 27667 consensus_queue.cc:1035] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:41:22.557436 27680 mvcc.cc:205] Tried to move back new op lower bound from 7316939090077511680 to 7316939089945767936. Current Snapshot: MvccSnapshot[applied={T|T < 7316939090077511680}]
I20260810 11:41:22.576058 26998 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:41:22.576392 26998 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60186:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:41:22.580778 26998 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Sending DeleteTablet for 3 replicas of tablet 42556d8e9227411299fc5e859d7ffe0d
I20260810 11:41:22.582144 27548 tablet_service.cc:1514] Processing DeleteTablet for tablet 42556d8e9227411299fc5e859d7ffe0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:22 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:42750
I20260810 11:41:22.582403 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 27072
I20260810 11:41:22.582587 27348 tablet_service.cc:1514] Processing DeleteTablet for tablet 42556d8e9227411299fc5e859d7ffe0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:22 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:39794
I20260810 11:41:22.582917 27165 tablet_service.cc:1514] Processing DeleteTablet for tablet 42556d8e9227411299fc5e859d7ffe0d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:22 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40788
I20260810 11:41:22.594455 27691 tablet_replica.cc:330] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: stopping tablet replica
I20260810 11:41:22.595911 27691 raft_consensus.cc:2229] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:41:22.596596 27691 raft_consensus.cc:2258] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:22.599409 27694 tablet_replica.cc:330] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: stopping tablet replica
I20260810 11:41:22.599436 27691 ts_tablet_manager.cc:1905] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:22.600390 27694 raft_consensus.cc:2229] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:41:22.601009 27694 raft_consensus.cc:2258] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:22.603266 27694 ts_tablet_manager.cc:1905] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:22.605475 27692 tablet_replica.cc:330] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: stopping tablet replica
I20260810 11:41:22.606016 27692 raft_consensus.cc:2229] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:41:22.606782 27692 raft_consensus.cc:2258] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:22.607462 27691 ts_tablet_manager.cc:1918] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:22.607688 27691 log.cc:1196] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/42556d8e9227411299fc5e859d7ffe0d
I20260810 11:41:22.608356 27691 ts_tablet_manager.cc:1939] T 42556d8e9227411299fc5e859d7ffe0d P dc59dc6710f64e1685fba3be4975b94b: Deleting consensus metadata
I20260810 11:41:22.610880 26975 catalog_manager.cc:4841] TS dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767): tablet 42556d8e9227411299fc5e859d7ffe0d (table test-table [id=3a6b7faf73234ef582f796869892b621]) successfully deleted
I20260810 11:41:22.612066 27694 ts_tablet_manager.cc:1918] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:22.612267 27694 log.cc:1196] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/42556d8e9227411299fc5e859d7ffe0d
Aug 10 11:41:22 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:41:22.612824 27694 ts_tablet_manager.cc:1939] T 42556d8e9227411299fc5e859d7ffe0d P 7292538215da4edba214fd1d1ce8e69d: Deleting consensus metadata
I20260810 11:41:22.613392 27692 ts_tablet_manager.cc:1905] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:22.615176 26975 catalog_manager.cc:4841] TS 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489): tablet 42556d8e9227411299fc5e859d7ffe0d (table test-table [id=3a6b7faf73234ef582f796869892b621]) successfully deleted
I20260810 11:41:22.621922 27692 ts_tablet_manager.cc:1918] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:22.622182 27692 log.cc:1196] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/42556d8e9227411299fc5e859d7ffe0d
I20260810 11:41:22.622725 27655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.583651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40804 (local address 127.3.240.193:33489)
0810 11:41:22.583891 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.583900 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.595619 (+ 11719us) server_negotiation.cc:407] Connection header received
0810 11:41:22.596722 (+ 1103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.596728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.596824 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.596908 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:22.601365 (+ 4457us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.602559 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.607048 (+ 4489us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.607718 (+ 670us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.607825 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.613897 (+ 6072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:22.613914 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:22.613930 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:22.613977 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:22.618007 (+ 4030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.619580 (+ 1573us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.619589 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.619593 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.619757 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.620261 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.620266 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.620270 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.620819 (+ 549us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:22.621133 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.621812 (+ 679us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.622063 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:41:22.623945 27692 ts_tablet_manager.cc:1939] T 42556d8e9227411299fc5e859d7ffe0d P 90b53b7fbc83441ea6b542a1a39dd792: Deleting consensus metadata
I20260810 11:41:22.624652 27246 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:22.626370 26975 catalog_manager.cc:4841] TS 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281): tablet 42556d8e9227411299fc5e859d7ffe0d (table test-table [id=3a6b7faf73234ef582f796869892b621]) successfully deleted
I20260810 11:41:22.721568 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 27072
I20260810 11:41:22.755113 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 27266
Aug 10 11:41:22 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:41:22.781024 27653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.756237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39816 (local address 127.3.240.194:38281)
0810 11:41:22.756428 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.756433 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.756452 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:41:22.756509 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.756512 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.756581 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.756668 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:22.758372 (+ 1704us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.759560 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.761899 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.762733 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.762933 (+ 200us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.773757 (+ 10824us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:22.773775 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:22.773800 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:22.773860 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:22.777039 (+ 3179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.777962 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.777976 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.777979 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.778276 (+ 297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.778810 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.778815 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.778818 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.779414 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:22.779859 (+ 445us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.780172 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.780476 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:41:22.781911 27435 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:22.903352 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 27266
I20260810 11:41:22.932467 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 27448
Aug 10 11:41:22 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:41:22.951048 27651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:22.933469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42762 (local address 127.3.240.195:40767)
0810 11:41:22.933659 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:22.933665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:22.933721 (+ 56us) server_negotiation.cc:407] Connection header received
0810 11:41:22.933823 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:22.933829 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:22.933925 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:22.934042 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:22.935820 (+ 1778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.936548 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.937940 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:22.938344 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:22.938451 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:22.944334 (+ 5883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:22.944357 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:22.944382 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:22.944437 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:22.947965 (+ 3528us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.948772 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.948780 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.948782 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.948922 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:22.949343 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:22.949347 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:22.949349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:22.949833 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:22.950075 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:22.950433 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 11:41:22.950602 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:41:22.951730 27628 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:23.046759 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 27448
I20260810 11:41:23.074843 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 26949
I20260810 11:41:23.076128 27033 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:23.197788 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 26949
I20260810 11:41:43.232718 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--webserver_interface=127.3.240.254
--webserver_port=39539
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:41:43.617632 27702 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:43.617938 27702 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:43.618112 27702 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:43.626127 27702 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 11:41:43.626233 27702 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 11:41:43.626274 27702 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 11:41:43.626319 27702 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:41:43.626452 27702 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:43.626567 27702 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:43.626621 27702 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:41:43.626672 27702 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:41:43.636987 27702 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=39539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:43.638317 27702 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:43.639721 27702 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:43.667261 27710 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:43.667407 27714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:43.667429 27711 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:43.668418 27702 server_base.cc:1034] running on GCE node
I20260810 11:41:43.669414 27702 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:43.671667 27702 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:43.672953 27702 hybrid_clock.cc:648] HybridClock initialized: now 1786362103672896 us; error 64 us; skew 500 ppm
I20260810 11:41:43.673626 27702 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:43 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362103, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:43.682861 27702 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:41:43.684767 27702 webserver.cc:460] Webserver started at http://127.3.240.254:39539/ using document root <none> and password file <none>
I20260810 11:41:43.685596 27702 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:43.685761 27702 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:43.693346 27702 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260810 11:41:43.697327 27722 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:43.699072 27702 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:41:43.699299 27702 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "077fb73607ea40f2889e37faedeb82c3"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "f894f34434b2f653b6db653c6359e8e3"
server_key_iv: "8c53fe6359f3e078cdcc4a0aaf3fab78"
server_key_version: "encryptionkey@0"
I20260810 11:41:43.700438 27702 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:43.732441 27702 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:43.739267 27702 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:43.739673 27702 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:43.764211 27702 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:36325
I20260810 11:41:43.764300 27797 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:36325 every 8 connection(s)
I20260810 11:41:43.765841 27702 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:41:43.770751 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 27702
I20260810 11:41:43.771723 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:33489
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=39325
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:41:43.775425 27800 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:41:43.778981 27800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap starting.
I20260810 11:41:43.801698 27800 log.cc:824] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:43.815527 27800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:41:43.816149 27800 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap complete.
I20260810 11:41:43.826193 27800 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:43.827116 27800 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Initialized, Role: FOLLOWER
I20260810 11:41:43.827739 27800 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:43.827998 27800 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:41:43.828138 27800 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:41:43.828330 27800 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 1 FOLLOWER]: Advancing to term 2
I20260810 11:41:43.833810 27800 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:43.834724 27800 leader_election.cc:302] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077fb73607ea40f2889e37faedeb82c3; no voters:
I20260810 11:41:43.835635 27800 leader_election.cc:288] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260810 11:41:43.835829 27805 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Leader election won for term 2
I20260810 11:41:43.837603 27805 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 LEADER]: Becoming Leader. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Running, Role: LEADER
I20260810 11:41:43.838274 27805 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:41:43.838697 27800 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:41:43.841224 27808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077fb73607ea40f2889e37faedeb82c3. Latest consensus state: current_term: 2 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:41:43.841746 27808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:43.842384 27806 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:41:43.842756 27806 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:41:43.850395 27813 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:41:43.861991 27813 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:41:43.863993 27813 catalog_manager.cc:1259] Loaded cluster ID: 3c0e3a89c88f451dae2c03058982b736
I20260810 11:41:43.864130 27813 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:41:43.871096 27813 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:41:43.873445 27813 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Loaded TSK: 0
I20260810 11:41:43.915968 27813 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Generated new TSK 1
I20260810 11:41:43.916440 27813 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 11:41:44.181468 27803 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:44.181836 27803 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:44.182011 27803 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:44.191380 27803 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:44.191686 27803 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:44.191905 27803 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:41:44.203915 27803 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:33489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=39325
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:44.205444 27803 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:44.206913 27803 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:44.223918 27842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:44.224057 27841 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:44.223999 27844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:44.224525 27803 server_base.cc:1034] running on GCE node
I20260810 11:41:44.225209 27803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:44.226253 27803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:44.227571 27803 hybrid_clock.cc:648] HybridClock initialized: now 1786362104227506 us; error 53 us; skew 500 ppm
I20260810 11:41:44.227949 27803 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:44 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:44.231233 27803 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:41:44.232892 27803 webserver.cc:460] Webserver started at http://127.3.240.193:39325/ using document root <none> and password file <none>
I20260810 11:41:44.233594 27803 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:44.233750 27803 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:44.240758 27803 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:41:44.244874 27856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.246389 27803 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:41:44.246572 27803 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "7292538215da4edba214fd1d1ce8e69d"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "ebf5f44ef71179a26de8c28bf1419c20"
server_key_iv: "7acab3ffb18b1ad9b02a94f4f6b063c2"
server_key_version: "encryptionkey@0"
I20260810 11:41:44.247650 27803 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:44.265945 27803 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:44.271381 27803 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:44.271626 27803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:44.272550 27803 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:44.273890 27803 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:44.273983 27803 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.274055 27803 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:44.274116 27803 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.309860 27803 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33489
I20260810 11:41:44.309957 28022 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33489 every 8 connection(s)
I20260810 11:41:44.311424 27803 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:41:44.321697 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 27803
I20260810 11:41:44.322526 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:38281
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=33159
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:41:44 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:44.338485 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:44.314611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:47389 (local address 127.3.240.254:36325)
0810 11:41:44.315508 (+ 897us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:44.315526 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:44.315952 (+ 426us) server_negotiation.cc:407] Connection header received
0810 11:41:44.318152 (+ 2200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:44.318164 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:44.318476 (+ 312us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:44.318851 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:44.320521 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.321935 (+ 1414us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.323742 (+ 1807us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.324851 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.325159 (+ 308us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:44.329919 (+ 4760us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:44.329928 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:44.329942 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:44.329989 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:44.333406 (+ 3417us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:44.334370 (+ 964us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:44.334378 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:44.334383 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:44.334562 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:44.335121 (+ 559us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:44.335126 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:44.335128 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:44.335646 (+ 518us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:44.335905 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:44.336750 (+ 845us) server_negotiation.cc:299] Negotiation successful
0810 11:41:44.337167 (+ 417us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":455,"threads_started":1}
I20260810 11:41:44.338485 28027 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:44.314786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:47389)
0810 11:41:44.315766 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:44.315805 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:44.317583 (+ 1778us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:44.319029 (+ 1446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:44.319043 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:44.319725 (+ 682us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:44.320266 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:44.320284 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.322118 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.322129 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:44.323565 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:44.323580 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.325295 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.325306 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:44.325653 (+ 347us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:44.326649 (+ 996us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:44.326678 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:44.329598 (+ 2920us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:44.333693 (+ 4095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:44.333704 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:44.333724 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:44.334170 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:44.334704 (+ 534us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:44.334712 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:44.334718 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:44.334975 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:44.335825 (+ 850us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:44.335835 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:44.336204 (+ 369us) client_negotiation.cc:769] Sending connection context
0810 11:41:44.336492 (+ 288us) client_negotiation.cc:240] Negotiation successful
0810 11:41:44.337082 (+ 590us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":378,"threads_started":1}
I20260810 11:41:44.342195 28024 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:44.342540 28024 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:44.343348 28024 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:44.345810 27746 ts_manager.cc:194] Registered new tserver with Master: 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:44.348091 27746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:47389
W20260810 11:41:44.688928 28030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:44.689518 28030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:44.689707 28030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:44.698781 28030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:44.699007 28030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:44.699218 28030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:41:44.709698 28030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=33159
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:44.711162 28030 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:44.712608 28030 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:44.726358 28035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:44.726392 28038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:44.726565 28036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:44.727496 28030 server_base.cc:1034] running on GCE node
I20260810 11:41:44.728538 28030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:44.729521 28030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:44.730785 28030 hybrid_clock.cc:648] HybridClock initialized: now 1786362104730713 us; error 53 us; skew 500 ppm
I20260810 11:41:44.731189 28030 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:44 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:44.734586 28030 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:41:44.736477 28030 webserver.cc:460] Webserver started at http://127.3.240.194:33159/ using document root <none> and password file <none>
I20260810 11:41:44.737210 28030 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:44.737382 28030 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:44.743800 28030 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:41:44.747638 28048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.749383 28030 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:41:44.749576 28030 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "90b53b7fbc83441ea6b542a1a39dd792"
format_stamp: "Formatted at 2026-08-10 11:41:21 on dist-test-slave-drl0"
server_key: "048375057b0a55e72c80f798f02e268e"
server_key_iv: "223c42e7f241c04f4219299f02048453"
server_key_version: "encryptionkey@0"
I20260810 11:41:44.750844 28030 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:44.781047 28030 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:44.786593 28030 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:44.787047 28030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:44.788388 28030 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:44.789927 28030 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:44.790045 28030 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.790138 28030 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:44.790207 28030 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:44.832114 28030 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38281
I20260810 11:41:44.832377 28204 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38281 every 8 connection(s)
I20260810 11:41:44.833823 28030 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:41:44.841944 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28030
I20260810 11:41:44.842613 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:40767
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=37017
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 10 11:41:44 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362104, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:44.860903 28026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:44.836788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49665 (local address 127.3.240.254:36325)
0810 11:41:44.837041 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:44.837049 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:44.838536 (+ 1487us) server_negotiation.cc:407] Connection header received
0810 11:41:44.841357 (+ 2821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:44.841367 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:44.841667 (+ 300us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:44.841814 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:44.843817 (+ 2003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.844994 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.847182 (+ 2188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.848190 (+ 1008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.848367 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:44.852624 (+ 4257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:44.852634 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:44.852643 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:44.852692 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:44.856639 (+ 3947us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:44.857824 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:44.857830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:44.857833 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:44.858180 (+ 347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:44.858786 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:44.858790 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:44.858792 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:44.859189 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:44.859308 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:44.860252 (+ 944us) server_negotiation.cc:299] Negotiation successful
0810 11:41:44.860443 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 11:41:44.862370 28207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:44.837125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:49665)
0810 11:41:44.838291 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:44.838342 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:44.840974 (+ 2632us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:44.842317 (+ 1343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:44.842326 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:44.843020 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:44.843607 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:44.843625 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.845227 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.845233 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:44.846804 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:44.846827 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:44.848510 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:44.848517 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:44.848759 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:44.849545 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:44.849570 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:44.852328 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:44.856807 (+ 4479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:44.856819 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:44.856839 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:44.857531 (+ 692us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:44.858335 (+ 804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:44.858343 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:44.858349 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:44.858613 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:44.859337 (+ 724us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:44.859349 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:44.859813 (+ 464us) client_negotiation.cc:769] Sending connection context
0810 11:41:44.860135 (+ 322us) client_negotiation.cc:240] Negotiation successful
0810 11:41:44.860777 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":405,"threads_started":1}
I20260810 11:41:44.864346 28205 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:44.864745 28205 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:44.865639 28205 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:44.867370 27746 ts_manager.cc:194] Registered new tserver with Master: 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281)
I20260810 11:41:44.868292 27746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:49665
W20260810 11:41:45.220112 28208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:41:45.220584 28208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:41:45.220825 28208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:41:45.229489 28208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:41:45.229607 28208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:41:45.229754 28208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:41:45.241536 28208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:40767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=37017
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:41:45.243057 28208 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:41:45.244383 28208 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:41:45.260018 28217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:45.259935 28219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:41:45.259975 28221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:41:45.261255 28208 server_base.cc:1034] running on GCE node
I20260810 11:41:45.261971 28208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:41:45.263270 28208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:41:45.264763 28208 hybrid_clock.cc:648] HybridClock initialized: now 1786362105264668 us; error 88 us; skew 500 ppm
I20260810 11:41:45.265292 28208 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:41:45 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:41:45.269258 28208 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:41:45.271310 28208 webserver.cc:460] Webserver started at http://127.3.240.195:37017/ using document root <none> and password file <none>
I20260810 11:41:45.272140 28208 fs_manager.cc:359] Metadata directory not provided
I20260810 11:41:45.272328 28208 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:41:45.278201 28208 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 11:41:45.281239 28229 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:45.283288 28208 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 11:41:45.283488 28208 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "dc59dc6710f64e1685fba3be4975b94b"
format_stamp: "Formatted at 2026-08-10 11:41:22 on dist-test-slave-drl0"
server_key: "3681706a76b737c455716b81f8e64c1e"
server_key_iv: "6269da1cc0bd4bea1595f09a95bfa1d4"
server_key_version: "encryptionkey@0"
I20260810 11:41:45.284678 28208 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:41:45.319685 28208 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:41:45.324602 28208 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:41:45.324940 28208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:41:45.326037 28208 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:41:45.327713 28208 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:41:45.327843 28208 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:45.328084 28208 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:41:45.328195 28208 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:41:45.351826 28024 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
I20260810 11:41:45.371480 28208 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40767
I20260810 11:41:45.371582 28381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40767 every 8 connection(s)
I20260810 11:41:45.372963 28208 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:41:45.374046 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28208
Aug 10 11:41:45 dist-test-slave-drl0 krb5kdc[26929](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362105, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:41:45.395042 28391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.375334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36214 (local address 127.3.240.254:36325)
0810 11:41:45.376730 (+ 1396us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.376735 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.376750 (+ 15us) server_negotiation.cc:407] Connection header received
0810 11:41:45.376811 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.376815 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.376879 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.377009 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:45.379162 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.379915 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.382082 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.382617 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.382824 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.386815 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:45.386827 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:45.386833 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:45.386878 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:45.390484 (+ 3606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.391356 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.391364 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.391368 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.391523 (+ 155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.392772 (+ 1249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.392779 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.392783 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.393343 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:45.393645 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.394001 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.394512 (+ 511us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1296,"thread_start_us":3427,"threads_started":1}
I20260810 11:41:45.400815 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.376269 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:57275 (local address 127.3.240.254:36325)
0810 11:41:45.379016 (+ 2747us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.379022 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.379045 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:41:45.380499 (+ 1454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.380503 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.380563 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.380645 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:45.382529 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.383640 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.386450 (+ 2810us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.387210 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.387372 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.392868 (+ 5496us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:45.392876 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:45.392883 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:45.392938 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:45.397168 (+ 4230us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.398145 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.398153 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.398159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.398309 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.398822 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.398829 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.398833 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.399175 (+ 342us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:45.399343 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.400099 (+ 756us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.400357 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2588,"thread_start_us":269,"threads_started":1}
I20260810 11:41:45.401795 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.377108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:57275)
0810 11:41:45.378129 (+ 1021us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.378180 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.380211 (+ 2031us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.380929 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.380941 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.381625 (+ 684us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:41:45.382333 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.382350 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.383867 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.383875 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.386253 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.386273 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.387475 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.387480 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.387683 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.388746 (+ 1063us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:41:45.388815 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:41:45.392245 (+ 3430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:41:45.397336 (+ 5091us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:45.397346 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:45.397363 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:45.397991 (+ 628us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:45.398455 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:41:45.398465 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:41:45.398468 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:41:45.398675 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:41:45.399338 (+ 663us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:41:45.399347 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:41:45.399688 (+ 341us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.399998 (+ 310us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.400543 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":671,"thread_start_us":415,"threads_started":1}
I20260810 11:41:45.403427 28383 heartbeater.cc:347] Connected to a master server at 127.3.240.254:36325
I20260810 11:41:45.403765 28383 heartbeater.cc:464] Registering TS with master...
I20260810 11:41:45.404587 28383 heartbeater.cc:511] Master 127.3.240.254:36325 requested a full tablet report, sending...
I20260810 11:41:45.406318 27746 ts_manager.cc:194] Registered new tserver with Master: dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767)
I20260810 11:41:45.407207 27746 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:57275
I20260810 11:41:45.411485 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:41:45.419997 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.412660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36226 (local address 127.3.240.254:36325)
0810 11:41:45.412968 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.412979 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.413444 (+ 465us) server_negotiation.cc:407] Connection header received
0810 11:41:45.413655 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.413660 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.413852 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.414024 (+ 172us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:41:45.414609 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.415691 (+ 1082us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.417418 (+ 1727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.417973 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.418127 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.418833 (+ 706us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:41:45.419395 (+ 562us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0810 11:41:45.419716 (+ 321us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:36226: token expired
Metrics: {"server-negotiator.queue_time_us":107}
W20260810 11:41:45.420128 28389 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:36226: token expired
W20260810 11:41:45.420745 27638 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.3.240.254:36325)
I20260810 11:41:45.434470 28389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.421224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36230 (local address 127.3.240.254:36325)
0810 11:41:45.421394 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.421398 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.421415 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:41:45.421496 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.421500 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.421557 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.421643 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:45.423118 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.423810 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.425317 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.426081 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.426225 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.428993 (+ 2768us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:45.428999 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:45.429002 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:45.429035 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:45.431742 (+ 2707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.432511 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.432516 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.432519 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.432724 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.433181 (+ 457us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.433186 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.433188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.433527 (+ 339us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:45.433745 (+ 218us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.434044 (+ 299us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.434196 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260810 11:41:45.438405 27746 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36230:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:41:45.446592 27746 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:41:45.477108 28398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.460782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:40767 (local address 127.0.0.1:41294)
0810 11:41:45.462378 (+ 1596us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.462436 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.462632 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.463524 (+ 892us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.463532 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.463565 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:45.465418 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.465439 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.469690 (+ 4251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.469699 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.473609 (+ 3910us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.473632 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.476261 (+ 2629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.476267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.476462 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.476501 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.476592 (+ 91us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.476726 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1040,"mutex_wait_us":52,"thread_start_us":736,"threads_started":1}
I20260810 11:41:45.477963 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.462836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.3.240.194:38281)
0810 11:41:45.463403 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.463412 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.463441 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:41:45.463541 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.463548 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.463740 (+ 192us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.463987 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:45.465146 (+ 1159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.467235 (+ 2089us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.474314 (+ 7079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.476268 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.476430 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.476662 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.477423 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.477601 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":387,"thread_start_us":658,"threads_started":1}
I20260810 11:41:45.478580 28400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.462333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38281 (local address 127.0.0.1:40870)
0810 11:41:45.462919 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.462959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.463065 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.464270 (+ 1205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.464278 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.464310 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:45.464887 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.464904 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.467452 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.467461 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.474007 (+ 6546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.474031 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.476854 (+ 2823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.476862 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.477105 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.477141 (+ 36us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.477265 (+ 124us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.477429 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":672,"threads_started":1}
I20260810 11:41:45.479394 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.461068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41294 (local address 127.3.240.195:40767)
0810 11:41:45.462573 (+ 1505us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.462581 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.462618 (+ 37us) server_negotiation.cc:407] Connection header received
0810 11:41:45.463004 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.463016 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.463272 (+ 256us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.465669 (+ 2397us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:45.466043 (+ 374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.469497 (+ 3454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.473893 (+ 4396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.476022 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.477927 (+ 1905us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.478586 (+ 659us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.478856 (+ 270us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.479052 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1327,"thread_start_us":1215,"threads_started":1}
I20260810 11:41:45.482613 28113 tablet_service.cc:1467] Processing CreateTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 (DEFAULT_TABLE table=test-table [id=e4fc062aa25547378aee859c79b7c83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:45.482980 28288 tablet_service.cc:1467] Processing CreateTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 (DEFAULT_TABLE table=test-table [id=e4fc062aa25547378aee859c79b7c83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:45.484192 28113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b422ebc4ed7e4d6f8c2a320602890c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:45.484953 28288 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b422ebc4ed7e4d6f8c2a320602890c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:45.485606 28404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.463338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35174)
0810 11:41:45.475905 (+ 12567us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.475925 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.476062 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.477148 (+ 1086us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.477153 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.477189 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:45.477551 (+ 362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.477563 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.480743 (+ 3180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.480749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.482821 (+ 2072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.482836 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.484774 (+ 1938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.484781 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.484993 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.485030 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.485125 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.485257 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12433,"thread_start_us":12265,"threads_started":1}
I20260810 11:41:45.485844 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.463627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35174 (local address 127.3.240.193:33489)
0810 11:41:45.476463 (+ 12836us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.476473 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.476504 (+ 31us) server_negotiation.cc:407] Connection header received
0810 11:41:45.476613 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.476622 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.476901 (+ 279us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.477128 (+ 227us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:45.477766 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.480558 (+ 2792us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.483073 (+ 2515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.484536 (+ 1463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.484684 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.484920 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.485192 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.485351 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12631,"thread_start_us":12539,"threads_started":1}
I20260810 11:41:45.489784 27927 tablet_service.cc:1467] Processing CreateTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 (DEFAULT_TABLE table=test-table [id=e4fc062aa25547378aee859c79b7c83d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:41:45.491240 27927 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b422ebc4ed7e4d6f8c2a320602890c54. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:41:45.497726 28406 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Bootstrap starting.
I20260810 11:41:45.502099 28406 tablet_bootstrap.cc:654] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:45.503547 28406 log.cc:824] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:45.503827 28407 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Bootstrap starting.
I20260810 11:41:45.507556 28410 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Bootstrap starting.
I20260810 11:41:45.509014 28407 tablet_bootstrap.cc:654] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:45.510597 28410 tablet_bootstrap.cc:654] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Neither blocks nor log segments found. Creating new log.
I20260810 11:41:45.510645 28407 log.cc:824] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:45.511790 28410 log.cc:824] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:41:45.514582 28406 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: No bootstrap required, opened a new log
I20260810 11:41:45.515130 28406 ts_tablet_manager.cc:1397] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.004s
I20260810 11:41:45.515440 28410 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: No bootstrap required, opened a new log
I20260810 11:41:45.515833 28410 ts_tablet_manager.cc:1397] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.002s
I20260810 11:41:45.515945 28407 tablet_bootstrap.cc:492] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: No bootstrap required, opened a new log
I20260810 11:41:45.516263 28407 ts_tablet_manager.cc:1397] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260810 11:41:45.523851 28410 raft_consensus.cc:348] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.524350 28410 raft_consensus.cc:374] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:45.524461 28410 raft_consensus.cc:729] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7292538215da4edba214fd1d1ce8e69d, State: Initialized, Role: FOLLOWER
I20260810 11:41:45.525213 28410 consensus_queue.cc:260] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.527231 28406 raft_consensus.cc:348] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.527698 28406 raft_consensus.cc:374] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:45.527801 28406 raft_consensus.cc:729] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b53b7fbc83441ea6b542a1a39dd792, State: Initialized, Role: FOLLOWER
I20260810 11:41:45.527602 28407 raft_consensus.cc:348] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.528018 28407 raft_consensus.cc:374] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:41:45.528131 28407 raft_consensus.cc:729] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc59dc6710f64e1685fba3be4975b94b, State: Initialized, Role: FOLLOWER
I20260810 11:41:45.528409 28410 ts_tablet_manager.cc:1428] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Time spent starting tablet: real 0.012s user 0.006s sys 0.005s
I20260810 11:41:45.528673 28407 consensus_queue.cc:260] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.528656 28406 consensus_queue.cc:260] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.530604 28407 ts_tablet_manager.cc:1428] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
I20260810 11:41:45.530733 28383 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
I20260810 11:41:45.532290 28205 heartbeater.cc:503] Master 127.3.240.254:36325 was elected leader, sending a full tablet report...
I20260810 11:41:45.532913 28406 ts_tablet_manager.cc:1428] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Time spent starting tablet: real 0.018s user 0.013s sys 0.004s
W20260810 11:41:45.564075 28025 tablet.cc:2369] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:41:45.586275 28206 tablet.cc:2369] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:41:45.625017 28385 tablet.cc:2369] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:41:45.795209 28418 raft_consensus.cc:482] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:41:45.795557 28418 raft_consensus.cc:504] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.797264 28418 leader_election.cc:288] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281), 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:45.805052 28392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.797592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:38281 (local address 127.3.240.195:39163)
0810 11:41:45.797807 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.797825 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.797965 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.798626 (+ 661us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.798632 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.798681 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:45.798993 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.799006 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.800319 (+ 1313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.800323 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.802405 (+ 2082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.802429 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.804202 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.804207 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.804429 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.804473 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.804585 (+ 112us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.804727 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"spinlock_wait_cycles":21248}
I20260810 11:41:45.805136 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.797988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:39163 (local address 127.3.240.194:38281)
0810 11:41:45.798245 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.798255 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.798283 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:41:45.798379 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.798384 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.798470 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.798581 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:45.799161 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.800148 (+ 987us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.802606 (+ 2458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.803993 (+ 1387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.804185 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.804265 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.804631 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.804728 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 11:41:45.805457 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.798422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:47219 (local address 127.3.240.193:33489)
0810 11:41:45.798649 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.798657 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.798677 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:41:45.798870 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.798873 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.798967 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.799037 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:41:45.799577 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.800516 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.802861 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.804254 (+ 1393us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.804443 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.804502 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.804862 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.804984 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 11:41:45.805754 28420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.798006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.3.240.195:47219)
0810 11:41:45.798608 (+ 602us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:45.798625 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:41:45.798730 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:41:45.799065 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:41:45.799071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:41:45.799100 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:41:45.799423 (+ 323us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.799430 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.800658 (+ 1228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.800663 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.802688 (+ 2025us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:41:45.802699 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.804499 (+ 1800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.804504 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:41:45.804683 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.804713 (+ 30us) client_negotiation.cc:769] Sending connection context
0810 11:41:45.804797 (+ 84us) client_negotiation.cc:240] Negotiation successful
0810 11:41:45.804934 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"mutex_wait_us":2,"thread_start_us":287,"threads_started":1}
I20260810 11:41:45.807057 28141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b422ebc4ed7e4d6f8c2a320602890c54" candidate_uuid: "dc59dc6710f64e1685fba3be4975b94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b53b7fbc83441ea6b542a1a39dd792" is_pre_election: true
I20260810 11:41:45.807071 27952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b422ebc4ed7e4d6f8c2a320602890c54" candidate_uuid: "dc59dc6710f64e1685fba3be4975b94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7292538215da4edba214fd1d1ce8e69d" is_pre_election: true
I20260810 11:41:45.807976 28141 raft_consensus.cc:2454] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc59dc6710f64e1685fba3be4975b94b in term 0.
I20260810 11:41:45.807976 27952 raft_consensus.cc:2454] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dc59dc6710f64e1685fba3be4975b94b in term 0.
I20260810 11:41:45.809008 28230 leader_election.cc:302] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b53b7fbc83441ea6b542a1a39dd792, dc59dc6710f64e1685fba3be4975b94b; no voters:
I20260810 11:41:45.809631 28418 raft_consensus.cc:2781] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:41:45.809829 28418 raft_consensus.cc:482] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:41:45.809962 28418 raft_consensus.cc:3037] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:45.812438 28418 raft_consensus.cc:504] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.813141 28418 leader_election.cc:288] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [CANDIDATE]: Term 1 election: Requested vote from peers 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281), 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489)
I20260810 11:41:45.813602 28141 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b422ebc4ed7e4d6f8c2a320602890c54" candidate_uuid: "dc59dc6710f64e1685fba3be4975b94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "90b53b7fbc83441ea6b542a1a39dd792"
I20260810 11:41:45.813717 27952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b422ebc4ed7e4d6f8c2a320602890c54" candidate_uuid: "dc59dc6710f64e1685fba3be4975b94b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7292538215da4edba214fd1d1ce8e69d"
I20260810 11:41:45.813853 28141 raft_consensus.cc:3037] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:45.814009 27952 raft_consensus.cc:3037] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:41:45.816411 28141 raft_consensus.cc:2454] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc59dc6710f64e1685fba3be4975b94b in term 1.
I20260810 11:41:45.816617 27952 raft_consensus.cc:2454] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dc59dc6710f64e1685fba3be4975b94b in term 1.
I20260810 11:41:45.816890 28230 leader_election.cc:302] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 90b53b7fbc83441ea6b542a1a39dd792, dc59dc6710f64e1685fba3be4975b94b; no voters:
I20260810 11:41:45.817170 28418 raft_consensus.cc:2781] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:41:45.817732 28418 raft_consensus.cc:686] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 1 LEADER]: Becoming Leader. State: Replica: dc59dc6710f64e1685fba3be4975b94b, State: Running, Role: LEADER
I20260810 11:41:45.818352 28418 consensus_queue.cc:237] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } }
I20260810 11:41:45.823824 27745 catalog_manager.cc:5495] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b reported cstate change: term changed from 0 to 1, leader changed from <none> to dc59dc6710f64e1685fba3be4975b94b (127.3.240.195). New cstate: current_term: 1 leader_uuid: "dc59dc6710f64e1685fba3be4975b94b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dc59dc6710f64e1685fba3be4975b94b" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 40767 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 } health_report { overall_health: UNKNOWN } } }
I20260810 11:41:45.844367 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.838855 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41296 (local address 127.3.240.195:40767)
0810 11:41:45.839095 (+ 240us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.839100 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.839119 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:41:45.839204 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.839207 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.839282 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.839427 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:41:45.839974 (+ 547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.840728 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.842186 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.842588 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.842732 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.843312 (+ 580us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:41:45.843767 (+ 455us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:41:45.843942 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.844024 (+ 82us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.844105 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:41:45.854666 28141 raft_consensus.cc:1261] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Refusing update from remote peer dc59dc6710f64e1685fba3be4975b94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:41:45.854923 27952 raft_consensus.cc:1261] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Refusing update from remote peer dc59dc6710f64e1685fba3be4975b94b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:41:45.856112 28422 consensus_queue.cc:1035] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "90b53b7fbc83441ea6b542a1a39dd792" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 38281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:41:45.856877 28418 consensus_queue.cc:1035] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7292538215da4edba214fd1d1ce8e69d" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 33489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:41:45.871385 28425 mvcc.cc:205] Tried to move back new op lower bound from 7316939185564938240 to 7316939185437126656. Current Snapshot: MvccSnapshot[applied={T|T < 7316939185564938240}]
I20260810 11:41:45.874675 28426 mvcc.cc:205] Tried to move back new op lower bound from 7316939185564938240 to 7316939185437126656. Current Snapshot: MvccSnapshot[applied={T|T < 7316939185564938240}]
I20260810 11:41:45.887522 27746 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:41:45.887813 27746 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36230:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:41:45.892249 27746 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Sending DeleteTablet for 3 replicas of tablet b422ebc4ed7e4d6f8c2a320602890c54
I20260810 11:41:45.893445 28288 tablet_service.cc:1514] Processing DeleteTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41294
I20260810 11:41:45.893728 27927 tablet_service.cc:1514] Processing DeleteTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:35174
I20260810 11:41:45.893834 28113 tablet_service.cc:1514] Processing DeleteTablet for tablet b422ebc4ed7e4d6f8c2a320602890c54 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:41:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:40870
I20260810 11:41:45.895172 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 27803
I20260810 11:41:45.904541 28439 tablet_replica.cc:330] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: stopping tablet replica
I20260810 11:41:45.905629 28439 raft_consensus.cc:2229] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:41:45.906843 28439 raft_consensus.cc:2258] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:45.907876 28437 tablet_replica.cc:330] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: stopping tablet replica
I20260810 11:41:45.908950 28437 raft_consensus.cc:2229] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:41:45.910590 28439 ts_tablet_manager.cc:1905] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:45.911361 28437 raft_consensus.cc:2258] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:45.914242 28437 ts_tablet_manager.cc:1905] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:45.915792 28438 tablet_replica.cc:330] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: stopping tablet replica
I20260810 11:41:45.916253 28438 raft_consensus.cc:2229] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:41:45.916790 28438 raft_consensus.cc:2258] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:41:45.924942 28438 ts_tablet_manager.cc:1905] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:41:45.925238 28439 ts_tablet_manager.cc:1918] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:45.925472 28439 log.cc:1196] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/b422ebc4ed7e4d6f8c2a320602890c54
I20260810 11:41:45.926110 28437 ts_tablet_manager.cc:1918] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:45.926143 28439 ts_tablet_manager.cc:1939] T b422ebc4ed7e4d6f8c2a320602890c54 P 90b53b7fbc83441ea6b542a1a39dd792: Deleting consensus metadata
I20260810 11:41:45.926312 28437 log.cc:1196] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/b422ebc4ed7e4d6f8c2a320602890c54
I20260810 11:41:45.926916 28437 ts_tablet_manager.cc:1939] T b422ebc4ed7e4d6f8c2a320602890c54 P dc59dc6710f64e1685fba3be4975b94b: Deleting consensus metadata
I20260810 11:41:45.928357 27723 catalog_manager.cc:4841] TS 90b53b7fbc83441ea6b542a1a39dd792 (127.3.240.194:38281): tablet b422ebc4ed7e4d6f8c2a320602890c54 (table test-table [id=e4fc062aa25547378aee859c79b7c83d]) successfully deleted
I20260810 11:41:45.928732 27723 catalog_manager.cc:4841] TS dc59dc6710f64e1685fba3be4975b94b (127.3.240.195:40767): tablet b422ebc4ed7e4d6f8c2a320602890c54 (table test-table [id=e4fc062aa25547378aee859c79b7c83d]) successfully deleted
I20260810 11:41:45.929747 28405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:45.899967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35176 (local address 127.3.240.193:33489)
0810 11:41:45.900235 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:45.900245 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:45.903604 (+ 3359us) server_negotiation.cc:407] Connection header received
0810 11:41:45.903825 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:45.903834 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:45.903960 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:45.904091 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:45.909981 (+ 5890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.911593 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.914225 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:45.915225 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:45.915343 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:45.919832 (+ 4489us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:45.919856 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:45.919880 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:45.919945 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:45.925589 (+ 5644us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.926709 (+ 1120us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.926718 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.926722 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.926963 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:45.927504 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:45.927509 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:45.927512 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:45.928008 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:45.928318 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:45.929001 (+ 683us) server_negotiation.cc:299] Negotiation successful
0810 11:41:45.929280 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260810 11:41:45.931351 28018 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:46.029170 28438 ts_tablet_manager.cc:1918] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:41:46.029490 28438 log.cc:1196] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/b422ebc4ed7e4d6f8c2a320602890c54
I20260810 11:41:46.029794 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 27803
I20260810 11:41:46.064180 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 28030
W20260810 11:41:46.064281 27723 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:33489 (error 108)
I20260810 11:41:46.066726 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.065809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35182)
0810 11:41:46.066291 (+ 482us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.066529 (+ 238us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":340,"thread_start_us":276,"threads_started":1}
W20260810 11:41:46.067034 27723 catalog_manager.cc:4584] TS 7292538215da4edba214fd1d1ce8e69d (127.3.240.193:33489): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet b422ebc4ed7e4d6f8c2a320602890c54: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
I20260810 11:41:46.084982 28401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.065690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40878 (local address 127.3.240.194:38281)
0810 11:41:46.065898 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:46.065904 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:46.065926 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:41:46.066024 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:46.066033 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:46.066138 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:46.066249 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:46.068510 (+ 2261us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:46.069234 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:46.071557 (+ 2323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:46.072111 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:46.072276 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:46.076320 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:46.076338 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:46.076354 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:46.076405 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:46.080609 (+ 4204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:46.082247 (+ 1638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:46.082255 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:46.082258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:46.082430 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:46.082907 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:46.082911 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:46.082913 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:46.083464 (+ 551us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:46.083702 (+ 238us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:46.084288 (+ 586us) server_negotiation.cc:299] Negotiation successful
0810 11:41:46.084495 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:41:46.085702 28196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:46.121477 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.120887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35192)
0810 11:41:46.121183 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.121300 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":170}
I20260810 11:41:46.122715 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.122275 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35204)
0810 11:41:46.122473 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.122559 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260810 11:41:46.183549 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.182994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35220)
0810 11:41:46.183241 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.183358 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 11:41:46.184791 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.184326 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35226)
0810 11:41:46.184505 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.184609 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20260810 11:41:46.192971 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 28030
I20260810 11:41:46.228889 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 28208
I20260810 11:41:46.249193 28399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.229836 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41302 (local address 127.3.240.195:40767)
0810 11:41:46.230039 (+ 203us) server_negotiation.cc:206] Beginning negotiation
0810 11:41:46.230043 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:41:46.230062 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:41:46.230198 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:41:46.230202 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:41:46.230287 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:41:46.230392 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:41:46.232508 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:46.233454 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:46.235843 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:41:46.236464 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:41:46.236633 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:41:46.240600 (+ 3967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:41:46.240611 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:41:46.240628 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:41:46.240672 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:41:46.245445 (+ 4773us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:46.246308 (+ 863us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:46.246322 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:46.246333 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:46.246557 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:41:46.247266 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:41:46.247308 (+ 42us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:41:46.247313 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:41:46.247823 (+ 510us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:41:46.248154 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0810 11:41:46.248521 (+ 367us) server_negotiation.cc:299] Negotiation successful
0810 11:41:46.248799 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 11:41:46.250164 28375 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:46.266346 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.265671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35228)
0810 11:41:46.265993 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.266143 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 11:41:46.267671 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.267178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35238)
0810 11:41:46.267409 (+ 231us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.267508 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111}
I20260810 11:41:46.384086 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 28208
I20260810 11:41:46.411623 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 27702
I20260810 11:41:46.413281 27791 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:41:46.539880 28444 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:41:46.539212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:33489 (local address 127.0.0.1:35248)
0810 11:41:46.539493 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0810 11:41:46.539621 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:33489: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 11:41:46.540289 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 27702
I20260810 11:42:06.571180 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--webserver_interface=127.3.240.254
--webserver_port=39539
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:36325
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:06.980356 28451 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:06.980693 28451 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:06.980868 28451 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:06.988678 28451 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260810 11:42:06.988834 28451 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260810 11:42:06.988932 28451 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260810 11:42:06.989034 28451 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:06.989142 28451 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:06.989267 28451 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:06.989362 28451 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:06.989447 28451 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:06.999704 28451 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:36325
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:36325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=39539
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:07.001077 28451 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:07.002488 28451 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:07.016425 28458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:07.016537 28463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:07.016491 28461 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:07.016533 28451 server_base.cc:1034] running on GCE node
I20260810 11:42:07.017843 28451 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:07.019336 28451 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:07.020612 28451 hybrid_clock.cc:648] HybridClock initialized: now 1786362127020509 us; error 80 us; skew 500 ppm
I20260810 11:42:07.020998 28451 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:07 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362127, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:07.024691 28451 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:42:07.026508 28451 webserver.cc:460] Webserver started at http://127.3.240.254:39539/ using document root <none> and password file <none>
I20260810 11:42:07.027382 28451 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:07.027522 28451 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:07.033288 28451 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:42:07.036908 28472 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:07.038797 28451 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:42:07.039251 28451 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "077fb73607ea40f2889e37faedeb82c3"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "f894f34434b2f653b6db653c6359e8e3"
server_key_iv: "8c53fe6359f3e078cdcc4a0aaf3fab78"
server_key_version: "encryptionkey@0"
I20260810 11:42:07.040283 28451 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:07.084007 28451 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:07.088824 28451 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:07.089141 28451 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:07.121754 28451 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:36325
I20260810 11:42:07.121793 28541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:36325 every 8 connection(s)
I20260810 11:42:07.123543 28451 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:07.126116 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28451
I20260810 11:42:07.127210 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:33489
--local_ip_for_outbound_sockets=127.3.240.193
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=39325
--webserver_interface=127.3.240.193
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:42:07.132715 28543 sys_catalog.cc:263] Verifying existing consensus state
I20260810 11:42:07.136224 28543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap starting.
I20260810 11:42:07.160979 28543 log.cc:824] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:07.178232 28543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260810 11:42:07.178800 28543 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Bootstrap complete.
I20260810 11:42:07.188849 28543 raft_consensus.cc:348] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:42:07.189766 28543 raft_consensus.cc:729] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Initialized, Role: FOLLOWER
I20260810 11:42:07.190471 28543 consensus_queue.cc:260] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:42:07.190761 28543 raft_consensus.cc:388] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:07.190912 28543 raft_consensus.cc:482] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:07.191167 28543 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 2 FOLLOWER]: Advancing to term 3
I20260810 11:42:07.195101 28543 raft_consensus.cc:504] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:42:07.195593 28543 leader_election.cc:302] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 077fb73607ea40f2889e37faedeb82c3; no voters:
I20260810 11:42:07.196348 28543 leader_election.cc:288] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260810 11:42:07.196498 28549 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 3 FOLLOWER]: Leader election won for term 3
I20260810 11:42:07.198343 28549 raft_consensus.cc:686] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [term 3 LEADER]: Becoming Leader. State: Replica: 077fb73607ea40f2889e37faedeb82c3, State: Running, Role: LEADER
I20260810 11:42:07.198851 28543 sys_catalog.cc:564] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:07.199007 28549 consensus_queue.cc:237] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } }
I20260810 11:42:07.201789 28555 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 077fb73607ea40f2889e37faedeb82c3. Latest consensus state: current_term: 3 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:42:07.202162 28555 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:07.202677 28554 sys_catalog.cc:455] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "077fb73607ea40f2889e37faedeb82c3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "077fb73607ea40f2889e37faedeb82c3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 36325 } } }
I20260810 11:42:07.203001 28554 sys_catalog.cc:458] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:07.213016 28562 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:07.222636 28562 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:07.224923 28562 catalog_manager.cc:1259] Loaded cluster ID: 3c0e3a89c88f451dae2c03058982b736
I20260810 11:42:07.225118 28562 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:07.229276 28562 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:07.231129 28562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Loaded TSK: 0
I20260810 11:42:07.232071 28562 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 077fb73607ea40f2889e37faedeb82c3: Loaded TSK: 1
F20260810 11:42:07.233872 28562 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fe544bd49ad google::LogMessage::Fail() at ??:0
@ 0x7fe544bd788a google::LogMessage::SendToLog() at ??:0
@ 0x7fe544bd44de google::LogMessage::Flush() at ??:0
@ 0x7fe544bd5a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fe5489387d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fe54892773e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fe55a6031c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fe55a602783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fe55a68e2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fe55a68e0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fe55a6d85e9 std::function<>::operator()() at ??:0
@ 0x7fe55a606cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fe55a604c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fe55a68c610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fe55a68c439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe547820345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fe54782391c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fe547823799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fe5477feeaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fe54aad86db start_thread at ??:0
@ 0x7fe543a6e61f clone at ??:0
W20260810 11:42:07.629968 28546 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:07.630263 28546 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:07.630363 28546 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:07.638422 28546 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:07.638540 28546 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:07.638653 28546 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:07.650823 28546 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:33489
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=39325
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:07.652211 28546 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:07.654814 28546 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:07.671223 28614 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:07.671386 28615 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:07.671325 28618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:07.671736 28546 server_base.cc:1034] running on GCE node
I20260810 11:42:07.672420 28546 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:07.673424 28546 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:07.674798 28546 hybrid_clock.cc:648] HybridClock initialized: now 1786362127674680 us; error 51 us; skew 500 ppm
I20260810 11:42:07.675366 28546 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:07 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362127, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:07.678874 28546 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:42:07.680861 28546 webserver.cc:460] Webserver started at http://127.3.240.193:39325/ using document root <none> and password file <none>
I20260810 11:42:07.681977 28546 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:07.682220 28546 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:07.689147 28546 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260810 11:42:07.693017 28627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:07.694772 28546 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:42:07.695117 28546 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "7292538215da4edba214fd1d1ce8e69d"
format_stamp: "Formatted at 2026-08-10 11:41:20 on dist-test-slave-drl0"
server_key: "ebf5f44ef71179a26de8c28bf1419c20"
server_key_iv: "7acab3ffb18b1ad9b02a94f4f6b063c2"
server_key_version: "encryptionkey@0"
I20260810 11:42:07.697031 28546 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:07.715847 28546 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:07.722726 28546 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:07.723065 28546 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:07.724243 28546 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:07.726270 28637 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
W20260810 11:42:07.732642 28637 ts_tablet_manager.cc:1852] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Tablet Manager startup: Rolling forward tablet deletion of type TABLET_DATA_DELETED
I20260810 11:42:07.733686 28637 ts_tablet_manager.cc:1905] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:07.741081 28637 ts_tablet_manager.cc:1918] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:07.741394 28637 ts_tablet_manager.cc:1939] T b422ebc4ed7e4d6f8c2a320602890c54 P 7292538215da4edba214fd1d1ce8e69d: Deleting consensus metadata
I20260810 11:42:07.741914 28546 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 0 live tablets)
I20260810 11:42:07.742059 28546 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.016s user 0.001s sys 0.000s
I20260810 11:42:07.742219 28546 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:07.742319 28546 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:07.789937 28546 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:33489
I20260810 11:42:07.790030 28783 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:33489 every 8 connection(s)
I20260810 11:42:07.791363 28546 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:07.797384 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:07.795357 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:55551)
0810 11:42:07.796244 (+ 887us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:07.796545 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":373,"threads_started":1}
W20260810 11:42:07.799933 28787 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:36325 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:36325: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
I20260810 11:42:07.800304 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28546
I20260810 11:42:07.801005 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:07.800591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:32903)
0810 11:42:07.800795 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:07.800872 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20260810 11:42:07.801129 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:38281
--local_ip_for_outbound_sockets=127.3.240.194
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=33159
--webserver_interface=127.3.240.194
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:42:07.802539 28789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:07.801733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:47273)
0810 11:42:07.802114 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:07.802329 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 11:42:07.803027 28787 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:42:08.196523 28791 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:08.196879 28791 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:08.197080 28791 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:08.204707 28791 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:08.204878 28791 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:08.205070 28791 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:08.214901 28791 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:38281
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=33159
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:08.216302 28791 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:08.217684 28791 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:08.232816 28799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:08.232752 28804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:08.232841 28798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:08.237100 28791 server_base.cc:1034] running on GCE node
I20260810 11:42:08.237775 28791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:08.239030 28791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:08.242422 28791 hybrid_clock.cc:648] HybridClock initialized: now 1786362128242323 us; error 74 us; skew 500 ppm
I20260810 11:42:08.242722 28791 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:08 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:08.247241 28791 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:42:08.249176 28791 webserver.cc:460] Webserver started at http://127.3.240.194:33159/ using document root <none> and password file <none>
I20260810 11:42:08.249989 28791 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:08.250133 28791 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:08.256179 28791 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:42:08.260110 28815 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.262110 28791 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:42:08.262319 28791 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "90b53b7fbc83441ea6b542a1a39dd792"
format_stamp: "Formatted at 2026-08-10 11:41:21 on dist-test-slave-drl0"
server_key: "048375057b0a55e72c80f798f02e268e"
server_key_iv: "223c42e7f241c04f4219299f02048453"
server_key_version: "encryptionkey@0"
I20260810 11:42:08.263890 28791 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:08.280118 28791 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:08.285650 28791 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:08.285964 28791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:08.287113 28791 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:08.288519 28791 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:08.288610 28791 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.288746 28791 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:08.288875 28791 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.329820 28791 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:38281
I20260810 11:42:08.329941 28969 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:38281 every 8 connection(s)
I20260810 11:42:08.331668 28791 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:08.339268 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.335768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:35209)
0810 11:42:08.337136 (+ 1368us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.337520 (+ 384us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":927,"thread_start_us":565,"threads_started":1}
I20260810 11:42:08.339255 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28791
I20260810 11:42:08.340161 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:40767
--local_ip_for_outbound_sockets=127.3.240.195
--tserver_master_addrs=127.3.240.254:36325
--webserver_port=37017
--webserver_interface=127.3.240.195
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:38027
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:08.343909 28970 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:36325 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:36325: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
I20260810 11:42:08.344990 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.344510 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:57589)
0810 11:42:08.344681 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.344867 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260810 11:42:08.346495 28972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.345759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:59075)
0810 11:42:08.346140 (+ 381us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.346307 (+ 167us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":221}
W20260810 11:42:08.346855 28970 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260810 11:42:08.714140 28975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:08.714447 28975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:08.714553 28975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:08.722568 28975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:08.722690 28975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:08.722815 28975 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:08.732511 28975 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38027
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:40767
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=37017
--tserver_master_addrs=127.3.240.254:36325
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:08.733738 28975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:08.735090 28975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:08.750285 28982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:08.750345 28984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:08.749892 28981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:08.751451 28975 server_base.cc:1034] running on GCE node
I20260810 11:42:08.752211 28975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:08.753260 28975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:08.754558 28975 hybrid_clock.cc:648] HybridClock initialized: now 1786362128754525 us; error 64 us; skew 500 ppm
I20260810 11:42:08.754913 28975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:08 dist-test-slave-drl0 krb5kdc[26929](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362128, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:08.758232 28975 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:42:08.759841 28975 webserver.cc:460] Webserver started at http://127.3.240.195:37017/ using document root <none> and password file <none>
I20260810 11:42:08.760564 28975 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:08.760722 28975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:08.766614 28975 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260810 11:42:08.769841 28998 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.771466 28975 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 11:42:08.771660 28975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "dc59dc6710f64e1685fba3be4975b94b"
format_stamp: "Formatted at 2026-08-10 11:41:22 on dist-test-slave-drl0"
server_key: "3681706a76b737c455716b81f8e64c1e"
server_key_iv: "6269da1cc0bd4bea1595f09a95bfa1d4"
server_key_version: "encryptionkey@0"
I20260810 11:42:08.772651 28975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:08.797520 28975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:08.802595 28975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:08.802879 28975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:08.804147 28975 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:08.804582 29006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.803623 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:46315)
0810 11:42:08.804201 (+ 578us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.804319 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":272,"threads_started":1}
I20260810 11:42:08.805642 28975 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:08.805796 28975 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.805925 28975 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:08.806016 28975 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:08.850042 28975 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:40767
I20260810 11:42:08.850201 29163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:40767 every 8 connection(s)
I20260810 11:42:08.851647 28975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:08.857028 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.854925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:51681)
0810 11:42:08.855774 (+ 849us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.856031 (+ 257us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":298,"threads_started":1}
I20260810 11:42:08.857252 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 28975
W20260810 11:42:08.857893 29164 heartbeater.cc:644] Failed to heartbeat to 127.3.240.254:36325 (0 consecutive failures): Network error: Failed to ping master at 127.3.240.254:36325: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
I20260810 11:42:08.859305 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.858706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:52573)
0810 11:42:08.859069 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.859186 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260810 11:42:08.860442 29166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:08.859965 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:38731)
0810 11:42:08.860150 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:08.860268 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260810 11:42:08.861449 29164 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260810 11:42:09.348610 29171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:09.347645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:54567)
0810 11:42:09.348187 (+ 542us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:09.348363 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":420,"thread_start_us":283,"threads_started":1}
I20260810 11:42:09.806684 29172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:09.805665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:51429)
0810 11:42:09.806414 (+ 749us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:09.806514 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":627,"thread_start_us":511,"threads_started":1}
I20260810 11:42:09.863266 29173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:09.862256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:60995)
0810 11:42:09.862875 (+ 619us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:09.863025 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":333,"threads_started":1}
I20260810 11:42:10.350263 29174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:10.349525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:47067)
0810 11:42:10.350009 (+ 484us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:10.350086 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":243,"threads_started":1}
I20260810 11:42:10.808619 29175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:10.807735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:52593)
0810 11:42:10.808361 (+ 626us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:10.808455 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":351,"threads_started":1}
I20260810 11:42:10.865140 29176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:10.864320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:36929)
0810 11:42:10.864883 (+ 563us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:10.864971 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":315,"threads_started":1}
I20260810 11:42:11.352006 29177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:11.351236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:57177)
0810 11:42:11.351740 (+ 504us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:11.351827 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":262,"threads_started":1}
I20260810 11:42:11.810307 29178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:11.809552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:36857)
0810 11:42:11.810038 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:11.810122 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":266,"threads_started":1}
I20260810 11:42:11.866919 29180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:11.866168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:50671)
0810 11:42:11.866646 (+ 478us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:11.866735 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":265,"threads_started":1}
I20260810 11:42:12.354210 29181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:12.353153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:57031)
0810 11:42:12.353780 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:12.353909 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":292,"threads_started":1}
I20260810 11:42:12.812367 29182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:12.811377 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:55421)
0810 11:42:12.812125 (+ 748us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:12.812203 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":643,"thread_start_us":437,"threads_started":1}
I20260810 11:42:12.868914 29184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:12.868033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:44547)
0810 11:42:12.868660 (+ 627us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:12.868742 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":404,"threads_started":1}
I20260810 11:42:13.356293 29186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:13.355561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:46597)
0810 11:42:13.356044 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:13.356124 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":261,"threads_started":1}
I20260810 11:42:13.814188 29187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:13.813467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:39237)
0810 11:42:13.813932 (+ 465us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:13.814014 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":247,"threads_started":1}
I20260810 11:42:13.871280 29188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:13.870422 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:42933)
0810 11:42:13.870995 (+ 573us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:13.871087 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":434,"thread_start_us":292,"threads_started":1}
I20260810 11:42:14.358661 29189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:14.357599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:59579)
0810 11:42:14.358364 (+ 765us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:14.358481 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":333,"threads_started":1}
I20260810 11:42:14.816577 29191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:14.815556 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:57919)
0810 11:42:14.816258 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:14.816363 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":426,"threads_started":1}
I20260810 11:42:14.873272 29192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:14.872421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:35589)
0810 11:42:14.872990 (+ 569us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:14.873103 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":340,"threads_started":1}
I20260810 11:42:15.360713 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:15.359814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:58011)
0810 11:42:15.360421 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:15.360500 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":298,"threads_started":1}
I20260810 11:42:15.818876 29195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:15.817854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:44109)
0810 11:42:15.818507 (+ 653us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:15.818635 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":373,"threads_started":1}
I20260810 11:42:15.876014 29197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:15.874726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:32969)
0810 11:42:15.875496 (+ 770us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:15.875617 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":436,"threads_started":1}
I20260810 11:42:16.362619 29199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:16.361850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:51575)
0810 11:42:16.362324 (+ 474us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:16.362413 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":255,"threads_started":1}
I20260810 11:42:16.820818 29200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:16.820018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:60799)
0810 11:42:16.820557 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:16.820652 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":284,"threads_started":1}
I20260810 11:42:16.878160 29202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:16.877259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:38041)
0810 11:42:16.877842 (+ 583us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:16.877965 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":295,"threads_started":1}
I20260810 11:42:17.364460 29204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:17.363618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:55155)
0810 11:42:17.364182 (+ 564us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:17.364274 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":279,"threads_started":1}
I20260810 11:42:17.823020 29205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:17.821935 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:55193)
0810 11:42:17.822582 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:17.822705 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":347,"threads_started":1}
I20260810 11:42:17.880764 29206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:17.879648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:52797)
0810 11:42:17.880445 (+ 797us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:17.880556 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":381,"threads_started":1}
I20260810 11:42:18.366335 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:18.365439 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:55501)
0810 11:42:18.366028 (+ 589us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:18.366133 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":328,"threads_started":1}
I20260810 11:42:18.825019 29211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:18.824199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:37161)
0810 11:42:18.824728 (+ 529us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:18.824828 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":307,"threads_started":1}
I20260810 11:42:18.882814 29212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:18.881946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:53261)
0810 11:42:18.882441 (+ 495us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:18.882529 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":271,"threads_started":1}
I20260810 11:42:19.368126 29214 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:19.367366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:49195)
0810 11:42:19.367854 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:19.367945 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":269,"threads_started":1}
I20260810 11:42:19.827040 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:19.826226 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:34281)
0810 11:42:19.826709 (+ 483us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:19.826788 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":281,"threads_started":1}
I20260810 11:42:19.884876 29216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:19.883816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:59917)
0810 11:42:19.884475 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:19.884640 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":344,"threads_started":1}
I20260810 11:42:20.369902 29217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:20.369077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:58137)
0810 11:42:20.369616 (+ 539us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:20.369737 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":251,"threads_started":1}
I20260810 11:42:20.828938 29218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:20.828119 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:36659)
0810 11:42:20.828567 (+ 448us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:20.828668 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":222,"threads_started":1}
I20260810 11:42:20.886847 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:20.886093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:53351)
0810 11:42:20.886573 (+ 480us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:20.886661 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":251,"threads_started":1}
I20260810 11:42:21.371740 29221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:21.370873 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:51705)
0810 11:42:21.371450 (+ 577us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:21.371548 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":340,"threads_started":1}
I20260810 11:42:21.831014 29222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:21.830093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:49921)
0810 11:42:21.830624 (+ 531us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:21.830728 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":402,"thread_start_us":303,"threads_started":1}
I20260810 11:42:21.888665 29223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:21.887889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:55087)
0810 11:42:21.888381 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:21.888474 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":389,"thread_start_us":251,"threads_started":1}
I20260810 11:42:22.373891 29225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:22.372720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:50617)
0810 11:42:22.373353 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:22.373606 (+ 253us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":321,"threads_started":1}
I20260810 11:42:22.833269 29226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:22.832380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:59091)
0810 11:42:22.832873 (+ 493us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:22.833084 (+ 211us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":381,"thread_start_us":256,"threads_started":1}
I20260810 11:42:22.890668 29228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:22.889674 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:42813)
0810 11:42:22.890160 (+ 486us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:22.890273 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":382,"thread_start_us":267,"threads_started":1}
I20260810 11:42:23.376869 29229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:23.375772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.194:60337)
0810 11:42:23.376443 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:23.376618 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":425,"threads_started":1}
I20260810 11:42:23.835355 29230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:23.834386 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.193:52109)
0810 11:42:23.835026 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:23.835143 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":296,"threads_started":1}
I20260810 11:42:23.864084 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 28546
I20260810 11:42:23.888952 29231 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:23.865373 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50206 (local address 127.3.240.193:33489)
0810 11:42:23.866145 (+ 772us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:23.866164 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:23.866223 (+ 59us) server_negotiation.cc:407] Connection header received
0810 11:42:23.868494 (+ 2271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:23.868505 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:23.869052 (+ 547us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:23.869415 (+ 363us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:23.871325 (+ 1910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:23.872566 (+ 1241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:23.873837 (+ 1271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:23.874502 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:23.874701 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:23.882176 (+ 7475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:23.882185 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:23.882201 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:23.882242 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:23.885310 (+ 3068us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:23.886208 (+ 898us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:23.886215 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:23.886222 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:23.886403 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:23.886790 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:23.886795 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:23.886798 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:23.887422 (+ 624us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:23.887730 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:23.888182 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 11:42:23.888640 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":364,"threads_started":1}
I20260810 11:42:23.890331 28779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:23.892894 29233 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:23.891999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:36325 (local address 127.3.240.195:34741)
0810 11:42:23.892495 (+ 496us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:23.892646 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.254:36325: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":349,"thread_start_us":262,"threads_started":1}
I20260810 11:42:23.988356 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 28546
I20260810 11:42:24.018410 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 28791
I20260810 11:42:24.040174 29235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:24.019858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56420 (local address 127.3.240.194:38281)
0810 11:42:24.020442 (+ 584us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:24.020457 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:24.020496 (+ 39us) server_negotiation.cc:407] Connection header received
0810 11:42:24.022454 (+ 1958us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:24.022465 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:24.022920 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:24.023308 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:24.024807 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:24.026380 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:24.027805 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:24.028502 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:24.028648 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:24.031882 (+ 3234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:24.031891 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:24.031906 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:24.031951 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:24.035734 (+ 3783us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:24.036676 (+ 942us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:24.036684 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:24.036691 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:24.036890 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:24.037390 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:24.037396 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:24.037399 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:24.038250 (+ 851us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:24.038596 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:24.039238 (+ 642us) server_negotiation.cc:299] Negotiation successful
0810 11:42:24.039793 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":416,"thread_start_us":288,"threads_started":1}
I20260810 11:42:24.042444 28965 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:24.133639 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 28791
I20260810 11:42:24.164011 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 28975
I20260810 11:42:24.186908 29237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:24.165350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57026 (local address 127.3.240.195:40767)
0810 11:42:24.166194 (+ 844us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:24.166214 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:24.166277 (+ 63us) server_negotiation.cc:407] Connection header received
0810 11:42:24.168528 (+ 2251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:24.168542 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:24.169237 (+ 695us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:24.169615 (+ 378us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:24.171724 (+ 2109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:24.173330 (+ 1606us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:24.175080 (+ 1750us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:24.175764 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:24.175946 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:24.179617 (+ 3671us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:24.179625 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:24.179640 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:24.179680 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:24.183520 (+ 3840us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:24.184288 (+ 768us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:24.184295 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:24.184301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:24.184483 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:24.184952 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:24.184956 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:24.184958 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:24.185512 (+ 554us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:24.185746 (+ 234us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:24.186235 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 11:42:24.186583 (+ 348us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":409,"threads_started":1}
I20260810 11:42:24.188412 29157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:24.280443 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 28975
2026-08-10T11:42:24Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66190 ms)
[----------] 30 tests from SecurityITest (334321 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:24 dist-test-slave-drl0 krb5kdc[29243](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:24 dist-test-slave-drl0 krb5kdc[29243](info): set up 2 sockets
Aug 10 11:42:24 dist-test-slave-drl0 krb5kdc[29243](info): commencing operation
krb5kdc: starting...
W20260810 11:42:25.673413 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.005s sys 0.046s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:25 dist-test-slave-drl0 krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:42:26.016000 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43941
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:43941
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:26.397847 29261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:26.398229 29261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:26.398423 29261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:26.406040 29261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:26.406208 29261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:26.406322 29261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:26.406590 29261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:26.406812 29261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:26.420604 29261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:43941
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:43941
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:26.422001 29261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:26.423414 29261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:26.437614 29271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:26.437569 29272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:26.437664 29274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:26.438903 29261 server_base.cc:1034] running on GCE node
I20260810 11:42:26.439733 29261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:26.441180 29261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:26.442602 29261 hybrid_clock.cc:648] HybridClock initialized: now 1786362146442531 us; error 36 us; skew 500 ppm
I20260810 11:42:26.443341 29261 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:26 dist-test-slave-drl0 krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362146, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:26.447642 29261 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:42:26.449364 29261 webserver.cc:460] Webserver started at http://127.3.240.254:42741/ using document root <none> and password file <none>
I20260810 11:42:26.450083 29261 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:26.450188 29261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:26.450548 29261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:26.453815 29261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "7a78a30a0ba94dc1bc5378c30e392941"
format_stamp: "Formatted at 2026-08-10 11:42:26 on dist-test-slave-drl0"
server_key: "0987b1a234321fc3e1a0fc9fbd627d2c"
server_key_iv: "db26a6b305a1150aba8514ec414be45e"
server_key_version: "encryptionkey@0"
I20260810 11:42:26.454656 29261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "7a78a30a0ba94dc1bc5378c30e392941"
format_stamp: "Formatted at 2026-08-10 11:42:26 on dist-test-slave-drl0"
server_key: "0987b1a234321fc3e1a0fc9fbd627d2c"
server_key_iv: "db26a6b305a1150aba8514ec414be45e"
server_key_version: "encryptionkey@0"
I20260810 11:42:26.461447 29261 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 11:42:26.465245 29284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:26.467414 29261 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 11:42:26.467756 29261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "7a78a30a0ba94dc1bc5378c30e392941"
format_stamp: "Formatted at 2026-08-10 11:42:26 on dist-test-slave-drl0"
server_key: "0987b1a234321fc3e1a0fc9fbd627d2c"
server_key_iv: "db26a6b305a1150aba8514ec414be45e"
server_key_version: "encryptionkey@0"
I20260810 11:42:26.468039 29261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:26.498726 29261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:26.503618 29261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:26.503942 29261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:26.526746 29350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:43941 every 8 connection(s)
I20260810 11:42:26.526823 29261 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:43941
I20260810 11:42:26.528808 29261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:26.530421 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 29261
I20260810 11:42:26.530711 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:26.531752 4035 external_mini_cluster.cc:1428] Setting key 23ad9b881e1835e9cb8ad6b597485706
I20260810 11:42:26.534698 29352 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 10 11:42:26 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:26.549746 29352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: Bootstrap starting.
I20260810 11:42:26.553589 29352 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:26.554966 29352 log.cc:824] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:26.559113 29352 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: No bootstrap required, opened a new log
I20260810 11:42:26.559188 29355 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:26.534310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59486 (local address 127.3.240.254:43941)
0810 11:42:26.535255 (+ 945us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:26.535267 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:26.535310 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:42:26.537131 (+ 1821us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:26.537154 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:26.537534 (+ 380us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:26.537908 (+ 374us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:26.540449 (+ 2541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:26.541871 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:26.543849 (+ 1978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:26.544681 (+ 832us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:26.544833 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:26.550403 (+ 5570us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:26.550413 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:26.550432 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:26.550482 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:26.553896 (+ 3414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:26.554779 (+ 883us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:26.554786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:26.554800 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:26.555057 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:26.555737 (+ 680us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:26.555746 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:26.555751 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:26.556590 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:26.556855 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:26.557761 (+ 906us) server_negotiation.cc:299] Negotiation successful
0810 11:42:26.558071 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":449,"threads_started":1}
I20260810 11:42:26.570601 29352 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } }
I20260810 11:42:26.571023 29352 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:26.571135 29352 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7a78a30a0ba94dc1bc5378c30e392941, State: Initialized, Role: FOLLOWER
I20260810 11:42:26.571770 29352 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } }
I20260810 11:42:26.572036 29352 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:26.572181 29352 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:26.572368 29352 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:26.574599 29352 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } }
I20260810 11:42:26.575203 29352 leader_election.cc:302] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7a78a30a0ba94dc1bc5378c30e392941; no voters:
I20260810 11:42:26.575899 29352 leader_election.cc:288] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:26.576090 29357 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:26.576825 29357 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [term 1 LEADER]: Becoming Leader. State: Replica: 7a78a30a0ba94dc1bc5378c30e392941, State: Running, Role: LEADER
I20260810 11:42:26.577346 29357 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } }
I20260810 11:42:26.578006 29352 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:26.581120 29360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7a78a30a0ba94dc1bc5378c30e392941. Latest consensus state: current_term: 1 leader_uuid: "7a78a30a0ba94dc1bc5378c30e392941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } } }
I20260810 11:42:26.581828 29360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:26.581305 29359 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7a78a30a0ba94dc1bc5378c30e392941" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7a78a30a0ba94dc1bc5378c30e392941" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 43941 } } }
I20260810 11:42:26.583601 29359 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:26.588205 29367 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:26.603624 29367 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:26.612704 29367 catalog_manager.cc:1347] Generated new cluster ID: 48d0c82720d84a7e8448064d914b16bd
I20260810 11:42:26.612835 29367 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:26.622509 29367 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:26.624430 29367 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:26.632325 29367 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: Generated new TSK 0
I20260810 11:42:26.633105 29367 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:42:26.723016 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:27.113598 29386 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:27.113901 29386 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:27.114005 29386 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:27.122463 29386 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:27.122574 29386 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:27.122691 29386 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:27.133662 29386 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:27.135119 29386 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:27.136392 29386 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:27.150274 29394 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:27.150349 29395 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:27.150405 29400 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:27.151326 29386 server_base.cc:1034] running on GCE node
I20260810 11:42:27.151952 29386 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:27.152920 29386 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:27.154188 29386 hybrid_clock.cc:648] HybridClock initialized: now 1786362147154137 us; error 64 us; skew 500 ppm
I20260810 11:42:27.154567 29386 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:27 dist-test-slave-drl0 krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:27.158493 29386 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:42:27.160090 29386 webserver.cc:460] Webserver started at http://127.3.240.193:40623/ using document root <none> and password file <none>
I20260810 11:42:27.160804 29386 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:27.160979 29386 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:27.161305 29386 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:27.164302 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "498361e242f042cf8297d1f83aa2c413"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "5479ae5e8db8cc7d7e72ee20e1df9426"
server_key_iv: "f832e2427377e2ae699d7ae79680b6ea"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.165117 29386 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "498361e242f042cf8297d1f83aa2c413"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "5479ae5e8db8cc7d7e72ee20e1df9426"
server_key_iv: "f832e2427377e2ae699d7ae79680b6ea"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.170475 29386 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260810 11:42:27.174525 29410 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.176194 29386 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:42:27.176406 29386 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "498361e242f042cf8297d1f83aa2c413"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "5479ae5e8db8cc7d7e72ee20e1df9426"
server_key_iv: "f832e2427377e2ae699d7ae79680b6ea"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.176625 29386 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:27.192354 29386 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:27.197274 29386 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:27.197717 29386 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:27.199007 29386 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:27.200733 29386 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:27.200819 29386 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.200977 29386 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:27.201063 29386 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.237438 29386 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36961
I20260810 11:42:27.237491 29574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36961 every 8 connection(s)
I20260810 11:42:27.238842 29386 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:27.240437 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 29386
I20260810 11:42:27.240657 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:27.241312 4035 external_mini_cluster.cc:1428] Setting key 7e538474a792e6575458c40acbf5be0c
Aug 10 11:42:27 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:27.262666 29578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:27.242575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:54517 (local address 127.3.240.254:43941)
0810 11:42:27.243418 (+ 843us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:27.243424 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:27.244812 (+ 1388us) server_negotiation.cc:407] Connection header received
0810 11:42:27.246774 (+ 1962us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:27.246782 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:27.246873 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:27.247029 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:27.248561 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.249708 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.251473 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.252248 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.252406 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:27.256464 (+ 4058us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:27.256473 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:27.256480 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:27.256525 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:27.259525 (+ 3000us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:27.260335 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:27.260347 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:27.260350 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:27.260475 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:27.260924 (+ 449us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:27.260927 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:27.260929 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:27.261198 (+ 269us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:27.261369 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:27.262053 (+ 684us) server_negotiation.cc:299] Negotiation successful
0810 11:42:27.262280 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":719,"thread_start_us":581,"threads_started":1}
I20260810 11:42:27.263715 29579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:27.243347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43941 (local address 127.3.240.193:54517)
0810 11:42:27.244472 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:27.244538 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:27.246509 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:27.247278 (+ 769us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:27.247286 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:27.247801 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:27.248383 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:27.248401 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.249883 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.249887 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:27.251313 (+ 1426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:27.251327 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.252495 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.252500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:27.252697 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:27.253432 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:27.253455 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:27.256239 (+ 2784us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:27.259695 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:27.259737 (+ 42us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:27.259759 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:27.260183 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:27.260590 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:27.260595 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:27.260599 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:27.260784 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:27.261333 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:27.261343 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:27.261673 (+ 330us) client_negotiation.cc:769] Sending connection context
0810 11:42:27.261930 (+ 257us) client_negotiation.cc:240] Negotiation successful
0810 11:42:27.262459 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":778,"thread_start_us":547,"threads_started":1}
I20260810 11:42:27.265334 29575 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43941
I20260810 11:42:27.265645 29575 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:27.266363 29575 heartbeater.cc:511] Master 127.3.240.254:43941 requested a full tablet report, sending...
I20260810 11:42:27.268752 29306 ts_manager.cc:194] Registered new tserver with Master: 498361e242f042cf8297d1f83aa2c413 (127.3.240.193:36961)
I20260810 11:42:27.270967 29306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:54517
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:42:27.318375 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:27.692328 29584 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:27.692616 29584 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:27.692718 29584 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:27.700127 29584 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:27.700242 29584 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:27.700356 29584 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:27.710107 29584 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:27.711388 29584 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:27.712683 29584 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:27.726668 29594 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:27.726853 29593 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:27.727250 29598 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:27.728675 29584 server_base.cc:1034] running on GCE node
I20260810 11:42:27.729321 29584 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:27.730341 29584 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:27.731657 29584 hybrid_clock.cc:648] HybridClock initialized: now 1786362147731536 us; error 57 us; skew 500 ppm
I20260810 11:42:27.732203 29584 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:27 dist-test-slave-drl0 krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:27.736034 29584 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:42:27.737753 29584 webserver.cc:460] Webserver started at http://127.3.240.194:43229/ using document root <none> and password file <none>
I20260810 11:42:27.738525 29584 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:27.738689 29584 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:27.739143 29584 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:27.742394 29584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "3f004e2132174053b719d6d846b63742"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "b84ce400ef60e76e7b54f5c446bb7b23"
server_key_iv: "4568511150c0f7e47feff8e91c2f6a74"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.743347 29584 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "3f004e2132174053b719d6d846b63742"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "b84ce400ef60e76e7b54f5c446bb7b23"
server_key_iv: "4568511150c0f7e47feff8e91c2f6a74"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.749475 29584 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:42:27.753942 29610 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.755625 29584 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 11:42:27.755831 29584 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "3f004e2132174053b719d6d846b63742"
format_stamp: "Formatted at 2026-08-10 11:42:27 on dist-test-slave-drl0"
server_key: "b84ce400ef60e76e7b54f5c446bb7b23"
server_key_iv: "4568511150c0f7e47feff8e91c2f6a74"
server_key_version: "encryptionkey@0"
I20260810 11:42:27.756052 29584 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:27.779346 29584 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:27.784269 29584 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:27.784534 29584 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:27.785774 29584 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:27.787555 29584 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:27.787643 29584 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.787737 29584 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:27.787783 29584 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:27.830677 29584 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:44833
I20260810 11:42:27.830724 29776 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:44833 every 8 connection(s)
I20260810 11:42:27.832445 29584 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:27.834604 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 29584
I20260810 11:42:27.834828 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:27.835775 4035 external_mini_cluster.cc:1428] Setting key 9266ce2ac54acd44517edfee6c915109
Aug 10 11:42:27 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362147, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:27.860303 29780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:27.835850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42293 (local address 127.3.240.254:43941)
0810 11:42:27.836495 (+ 645us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:27.836503 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:27.837961 (+ 1458us) server_negotiation.cc:407] Connection header received
0810 11:42:27.840576 (+ 2615us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:27.840582 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:27.840658 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:27.840780 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:27.843153 (+ 2373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.843926 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.847058 (+ 3132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.847521 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.847634 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:27.852746 (+ 5112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:27.852753 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:27.852757 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:27.852804 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:27.856207 (+ 3403us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:27.857372 (+ 1165us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:27.857377 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:27.857379 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:27.857528 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:27.858083 (+ 555us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:27.858087 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:27.858089 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:27.858418 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:27.858541 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:27.859698 (+ 1157us) server_negotiation.cc:299] Negotiation successful
0810 11:42:27.859909 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":470,"thread_start_us":302,"threads_started":1}
I20260810 11:42:27.861753 29781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:27.836359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43941 (local address 127.3.240.194:42293)
0810 11:42:27.837749 (+ 1390us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:27.837811 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:27.840217 (+ 2406us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:27.841102 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:27.841118 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:27.841998 (+ 880us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:27.842878 (+ 880us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:27.842908 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.844259 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.844269 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:27.846843 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:27.846869 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:27.847795 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:27.847805 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:27.848166 (+ 361us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:27.848975 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:27.849019 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:27.852361 (+ 3342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:27.856430 (+ 4069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:27.856443 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:27.856466 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:27.857201 (+ 735us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:27.857647 (+ 446us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:27.857655 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:27.857661 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:27.857919 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:27.858557 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:27.858569 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:27.859119 (+ 550us) client_negotiation.cc:769] Sending connection context
0810 11:42:27.859493 (+ 374us) client_negotiation.cc:240] Negotiation successful
0810 11:42:27.860153 (+ 660us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1088,"thread_start_us":842,"threads_started":1}
I20260810 11:42:27.863320 29777 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43941
I20260810 11:42:27.863888 29777 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:27.864974 29777 heartbeater.cc:511] Master 127.3.240.254:43941 requested a full tablet report, sending...
I20260810 11:42:27.867208 29306 ts_manager.cc:194] Registered new tserver with Master: 3f004e2132174053b719d6d846b63742 (127.3.240.194:44833)
I20260810 11:42:27.868319 29306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:42293
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:42:27.919895 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35989
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:42:28.274454 29575 heartbeater.cc:503] Master 127.3.240.254:43941 was elected leader, sending a full tablet report...
W20260810 11:42:28.322619 29785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:28.322916 29785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:28.323150 29785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:28.330823 29785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:28.331014 29785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:28.331215 29785 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:28.341183 29785 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35989
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:43941
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:28.342536 29785 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:28.343981 29785 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:28.358837 29800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:28.358901 29794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:28.359081 29797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:28.360528 29785 server_base.cc:1034] running on GCE node
I20260810 11:42:28.361285 29785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:28.362336 29785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:28.363615 29785 hybrid_clock.cc:648] HybridClock initialized: now 1786362148363485 us; error 131 us; skew 500 ppm
I20260810 11:42:28.364120 29785 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:28 dist-test-slave-drl0 krb5kdc[29243](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:28.367939 29785 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:42:28.369943 29785 webserver.cc:460] Webserver started at http://127.3.240.195:43643/ using document root <none> and password file <none>
I20260810 11:42:28.370985 29785 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:28.371169 29785 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:28.371572 29785 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:28.375351 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "a4c3fa2ac0bc4d9d83586422c50c174a"
format_stamp: "Formatted at 2026-08-10 11:42:28 on dist-test-slave-drl0"
server_key: "765e615d550c16bef44417f9f7fba7bd"
server_key_iv: "b1a81131e4f0e7e409abd607f9fb1363"
server_key_version: "encryptionkey@0"
I20260810 11:42:28.376552 29785 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "a4c3fa2ac0bc4d9d83586422c50c174a"
format_stamp: "Formatted at 2026-08-10 11:42:28 on dist-test-slave-drl0"
server_key: "765e615d550c16bef44417f9f7fba7bd"
server_key_iv: "b1a81131e4f0e7e409abd607f9fb1363"
server_key_version: "encryptionkey@0"
I20260810 11:42:28.382820 29785 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260810 11:42:28.386832 29809 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:28.388842 29785 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:42:28.389176 29785 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "a4c3fa2ac0bc4d9d83586422c50c174a"
format_stamp: "Formatted at 2026-08-10 11:42:28 on dist-test-slave-drl0"
server_key: "765e615d550c16bef44417f9f7fba7bd"
server_key_iv: "b1a81131e4f0e7e409abd607f9fb1363"
server_key_version: "encryptionkey@0"
I20260810 11:42:28.389436 29785 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:28.406785 29785 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:28.412674 29785 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:28.412992 29785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:28.414449 29785 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:28.416137 29785 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:28.416261 29785 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:28.416404 29785 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:28.416486 29785 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:28.458884 29785 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34957
I20260810 11:42:28.458978 29971 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34957 every 8 connection(s)
I20260810 11:42:28.460266 29785 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:28.469774 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 29785
I20260810 11:42:28.469981 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:28.470463 4035 external_mini_cluster.cc:1428] Setting key 5c744b777f263c94de6e3dd3ddd18d97
Aug 10 11:42:28 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362148, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:28.481865 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.463129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:42603 (local address 127.3.240.254:43941)
0810 11:42:28.463611 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.463616 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.464820 (+ 1204us) server_negotiation.cc:407] Connection header received
0810 11:42:28.466759 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.466765 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.466857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.467004 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:28.468502 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.469363 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.471392 (+ 2029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.471821 (+ 429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.471975 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.475811 (+ 3836us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:28.475816 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:28.475820 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:28.475851 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:28.478628 (+ 2777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.479456 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.479460 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.479463 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.479582 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.480035 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.480044 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.480046 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.480304 (+ 258us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:28.480483 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.481244 (+ 761us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.481480 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":364,"thread_start_us":373,"threads_started":1}
I20260810 11:42:28.482781 29976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.463862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:43941 (local address 127.3.240.195:42603)
0810 11:42:28.464628 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.464688 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.466488 (+ 1800us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.467232 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.467239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.467748 (+ 509us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:28.468334 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.468351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.469523 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.469527 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.471194 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.471213 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.472054 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.472062 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.472350 (+ 288us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.473218 (+ 868us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:28.473242 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:28.475587 (+ 2345us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:28.478782 (+ 3195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:28.478791 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:28.478806 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:28.479310 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:28.479699 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:28.479703 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:28.479707 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:28.479902 (+ 195us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:28.480437 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:28.480447 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:28.480786 (+ 339us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.481133 (+ 347us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.481643 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":315,"threads_started":1}
I20260810 11:42:28.484613 29973 heartbeater.cc:347] Connected to a master server at 127.3.240.254:43941
I20260810 11:42:28.484932 29973 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:28.485702 29973 heartbeater.cc:511] Master 127.3.240.254:43941 requested a full tablet report, sending...
I20260810 11:42:28.487218 29306 ts_manager.cc:194] Registered new tserver with Master: a4c3fa2ac0bc4d9d83586422c50c174a (127.3.240.195:34957)
I20260810 11:42:28.488122 29306 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:42603
I20260810 11:42:28.498340 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:28.517449 29975 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.503739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59502 (local address 127.3.240.254:43941)
0810 11:42:28.503978 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.503988 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.504470 (+ 482us) server_negotiation.cc:407] Connection header received
0810 11:42:28.504529 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.504533 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.504596 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.504680 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:28.506356 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.507266 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.508638 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.509057 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.509163 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.511752 (+ 2589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:28.511758 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:28.511763 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:28.511797 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:28.514550 (+ 2753us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.515361 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.515365 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.515368 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.515492 (+ 124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.515992 (+ 500us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.515996 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.515998 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.516356 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:28.516560 (+ 204us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.516896 (+ 336us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.517064 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:42:28.522697 29306 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:28.531100 29306 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:28.556629 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.547566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.3.240.194:44833)
0810 11:42:28.549026 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.549034 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.549063 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:42:28.549173 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.549184 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.549429 (+ 245us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.549702 (+ 273us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:28.550409 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.551890 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.554176 (+ 2286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.555324 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.555484 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.555699 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.556012 (+ 313us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.556173 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":615,"threads_started":1}
I20260810 11:42:28.556972 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.547228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:44833 (local address 127.0.0.1:43026)
0810 11:42:28.547835 (+ 607us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.547853 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.547948 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.549710 (+ 1762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.549717 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.549748 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:28.550175 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.550188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.552020 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.552028 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.553989 (+ 1961us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.554003 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.555561 (+ 1558us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.555567 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.555777 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.555815 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.555957 (+ 142us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.556150 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":322,"threads_started":1}
I20260810 11:42:28.558873 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.546285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56988 (local address 127.3.240.193:36961)
0810 11:42:28.547061 (+ 776us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.547070 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.547479 (+ 409us) server_negotiation.cc:407] Connection header received
0810 11:42:28.547925 (+ 446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.547934 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.548196 (+ 262us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.548389 (+ 193us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:28.549865 (+ 1476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.551572 (+ 1707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.554459 (+ 2887us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.556364 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.556538 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.556780 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.558301 (+ 1521us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.558506 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":560,"thread_start_us":354,"threads_started":1}
I20260810 11:42:28.559672 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.545909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36961 (local address 127.0.0.1:56988)
0810 11:42:28.547328 (+ 1419us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.547365 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.547658 (+ 293us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.549158 (+ 1500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.549164 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.549193 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:28.549613 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.549626 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.551714 (+ 2088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.551723 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.554280 (+ 2557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.554294 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.557820 (+ 3526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.557832 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.558027 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.558067 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.559040 (+ 973us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.559199 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":664,"mutex_wait_us":32,"thread_start_us":512,"threads_started":1}
I20260810 11:42:28.560266 29996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.547925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34957 (local address 127.0.0.1:46248)
0810 11:42:28.548821 (+ 896us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.548836 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.548961 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.550908 (+ 1947us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.550912 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.550961 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:28.551349 (+ 388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.551358 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.554114 (+ 2756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.554119 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.557347 (+ 3228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.557360 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.559404 (+ 2044us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.559411 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.559604 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.559643 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.559748 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.559887 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":601,"threads_started":1}
I20260810 11:42:28.560753 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.548877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46248 (local address 127.3.240.195:34957)
0810 11:42:28.550492 (+ 1615us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.550498 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.550519 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:42:28.550589 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.550594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.550726 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.550917 (+ 191us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:28.551520 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.553967 (+ 2447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.557513 (+ 3546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.558870 (+ 1357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.559244 (+ 374us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.559497 (+ 253us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.560145 (+ 648us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.560301 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1451,"thread_start_us":1278,"threads_started":1}
I20260810 11:42:28.561600 29681 tablet_service.cc:1467] Processing CreateTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 (DEFAULT_TABLE table=test-table [id=85bdef50ba1440338f53094e7398c9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:28.563127 29481 tablet_service.cc:1467] Processing CreateTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 (DEFAULT_TABLE table=test-table [id=85bdef50ba1440338f53094e7398c9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:28.563963 29681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66431ea1b6b43bf871842dfb33f34f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:28.564378 29481 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66431ea1b6b43bf871842dfb33f34f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:28.564340 29876 tablet_service.cc:1467] Processing CreateTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 (DEFAULT_TABLE table=test-table [id=85bdef50ba1440338f53094e7398c9a6]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:28.565687 29876 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b66431ea1b6b43bf871842dfb33f34f7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:28.576428 29999 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Bootstrap starting.
I20260810 11:42:28.579743 29999 tablet_bootstrap.cc:654] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:28.580915 29999 log.cc:824] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:28.583127 30002 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Bootstrap starting.
I20260810 11:42:28.585534 30001 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Bootstrap starting.
I20260810 11:42:28.587491 29999 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: No bootstrap required, opened a new log
I20260810 11:42:28.587783 29999 ts_tablet_manager.cc:1397] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20260810 11:42:28.589772 30002 tablet_bootstrap.cc:654] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:28.590312 30001 tablet_bootstrap.cc:654] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:28.591312 30002 log.cc:824] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:28.591658 30001 log.cc:824] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:28.594657 30002 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: No bootstrap required, opened a new log
I20260810 11:42:28.594846 30001 tablet_bootstrap.cc:492] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: No bootstrap required, opened a new log
I20260810 11:42:28.595017 30002 ts_tablet_manager.cc:1397] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260810 11:42:28.595145 30001 ts_tablet_manager.cc:1397] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Time spent bootstrapping tablet: real 0.010s user 0.009s sys 0.000s
I20260810 11:42:28.597492 29999 raft_consensus.cc:348] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.597945 29999 raft_consensus.cc:374] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:28.598047 29999 raft_consensus.cc:729] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 498361e242f042cf8297d1f83aa2c413, State: Initialized, Role: FOLLOWER
I20260810 11:42:28.598667 29999 consensus_queue.cc:260] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.600627 29999 ts_tablet_manager.cc:1428] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
I20260810 11:42:28.605461 30002 raft_consensus.cc:348] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.605952 30002 raft_consensus.cc:374] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:28.606083 30002 raft_consensus.cc:729] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3f004e2132174053b719d6d846b63742, State: Initialized, Role: FOLLOWER
I20260810 11:42:28.605844 30001 raft_consensus.cc:348] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.606338 30001 raft_consensus.cc:374] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:28.606468 30001 raft_consensus.cc:729] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4c3fa2ac0bc4d9d83586422c50c174a, State: Initialized, Role: FOLLOWER
I20260810 11:42:28.606914 30002 consensus_queue.cc:260] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.607156 30001 consensus_queue.cc:260] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.608317 29973 heartbeater.cc:503] Master 127.3.240.254:43941 was elected leader, sending a full tablet report...
I20260810 11:42:28.608323 29777 heartbeater.cc:503] Master 127.3.240.254:43941 was elected leader, sending a full tablet report...
I20260810 11:42:28.609001 30001 ts_tablet_manager.cc:1428] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Time spent starting tablet: real 0.014s user 0.009s sys 0.004s
I20260810 11:42:28.609001 30002 ts_tablet_manager.cc:1428] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
W20260810 11:42:28.712162 29974 tablet.cc:2369] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:28.712744 30009 raft_consensus.cc:482] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:28.713202 30009 raft_consensus.cc:504] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.715675 30009 leader_election.cc:288] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 498361e242f042cf8297d1f83aa2c413 (127.3.240.193:36961), a4c3fa2ac0bc4d9d83586422c50c174a (127.3.240.195:34957)
I20260810 11:42:28.724375 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.716196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:52291 (local address 127.3.240.193:36961)
0810 11:42:28.716463 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.716472 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.717147 (+ 675us) server_negotiation.cc:407] Connection header received
0810 11:42:28.717376 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.717384 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.717479 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.717580 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:28.718266 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.719626 (+ 1360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.721848 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.723152 (+ 1304us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.723286 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.723357 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.724067 (+ 710us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.724174 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:42:28.724439 30010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.716181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36961 (local address 127.3.240.194:52291)
0810 11:42:28.716971 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.717011 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.717218 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.717637 (+ 419us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.717647 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.717681 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:28.718066 (+ 385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.718077 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.719768 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.719774 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.721664 (+ 1890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.721677 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.723560 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.723569 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.723759 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.723799 (+ 40us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.723924 (+ 125us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.724062 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":424,"threads_started":1}
I20260810 11:42:28.725554 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.717264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:44147 (local address 127.3.240.195:34957)
0810 11:42:28.717530 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.717536 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.718259 (+ 723us) server_negotiation.cc:407] Connection header received
0810 11:42:28.718515 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.718519 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.718580 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.718668 (+ 88us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:28.719606 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.720800 (+ 1194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.723192 (+ 2392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.724636 (+ 1444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.724824 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.724909 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.725177 (+ 268us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.725305 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260810 11:42:28.725611 30011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.717061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34957 (local address 127.3.240.194:44147)
0810 11:42:28.718053 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:28.718073 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:28.718311 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:28.718740 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:28.718745 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:28.718775 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:28.719413 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.719430 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.721000 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.721004 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.722858 (+ 1854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:28.722874 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.724858 (+ 1984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.724867 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:28.725005 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.725029 (+ 24us) client_negotiation.cc:769] Sending connection context
0810 11:42:28.725149 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 11:42:28.725290 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":768,"thread_start_us":396,"threads_started":1}
I20260810 11:42:28.725553 29510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66431ea1b6b43bf871842dfb33f34f7" candidate_uuid: "3f004e2132174053b719d6d846b63742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498361e242f042cf8297d1f83aa2c413" is_pre_election: true
I20260810 11:42:28.726202 29510 raft_consensus.cc:2454] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f004e2132174053b719d6d846b63742 in term 0.
I20260810 11:42:28.726583 29905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66431ea1b6b43bf871842dfb33f34f7" candidate_uuid: "3f004e2132174053b719d6d846b63742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" is_pre_election: true
I20260810 11:42:28.726979 29616 leader_election.cc:302] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f004e2132174053b719d6d846b63742, 498361e242f042cf8297d1f83aa2c413; no voters:
I20260810 11:42:28.727137 29905 raft_consensus.cc:2454] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3f004e2132174053b719d6d846b63742 in term 0.
I20260810 11:42:28.727468 30009 raft_consensus.cc:2781] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:28.727632 30009 raft_consensus.cc:482] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:28.727784 30009 raft_consensus.cc:3037] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:28.730131 30009 raft_consensus.cc:504] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.730754 30009 leader_election.cc:288] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [CANDIDATE]: Term 1 election: Requested vote from peers 498361e242f042cf8297d1f83aa2c413 (127.3.240.193:36961), a4c3fa2ac0bc4d9d83586422c50c174a (127.3.240.195:34957)
I20260810 11:42:28.731457 29510 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66431ea1b6b43bf871842dfb33f34f7" candidate_uuid: "3f004e2132174053b719d6d846b63742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "498361e242f042cf8297d1f83aa2c413"
I20260810 11:42:28.731664 29510 raft_consensus.cc:3037] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:28.731654 29905 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b66431ea1b6b43bf871842dfb33f34f7" candidate_uuid: "3f004e2132174053b719d6d846b63742" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a"
I20260810 11:42:28.731860 29905 raft_consensus.cc:3037] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:28.733935 29510 raft_consensus.cc:2454] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f004e2132174053b719d6d846b63742 in term 1.
I20260810 11:42:28.734282 29905 raft_consensus.cc:2454] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3f004e2132174053b719d6d846b63742 in term 1.
I20260810 11:42:28.734369 29616 leader_election.cc:302] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3f004e2132174053b719d6d846b63742, 498361e242f042cf8297d1f83aa2c413; no voters:
I20260810 11:42:28.734853 30009 raft_consensus.cc:2781] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:28.735441 30009 raft_consensus.cc:686] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 1 LEADER]: Becoming Leader. State: Replica: 3f004e2132174053b719d6d846b63742, State: Running, Role: LEADER
I20260810 11:42:28.736065 30009 consensus_queue.cc:237] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } }
I20260810 11:42:28.741420 29305 catalog_manager.cc:5495] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 reported cstate change: term changed from 0 to 1, leader changed from <none> to 3f004e2132174053b719d6d846b63742 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "3f004e2132174053b719d6d846b63742" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "3f004e2132174053b719d6d846b63742" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 44833 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 } health_report { overall_health: UNKNOWN } } }
W20260810 11:42:28.742019 29576 tablet.cc:2369] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:28.801607 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.794558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43038 (local address 127.3.240.194:44833)
0810 11:42:28.794797 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.794802 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.794819 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:42:28.794911 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.794914 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.795008 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.795137 (+ 129us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:28.795859 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.796598 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.798492 (+ 1894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.798908 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.799033 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.799955 (+ 922us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:28.800697 (+ 742us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:28.800812 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.801235 (+ 423us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.801323 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260810 11:42:28.812206 29905 raft_consensus.cc:1261] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 1 FOLLOWER]: Refusing update from remote peer 3f004e2132174053b719d6d846b63742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:28.812458 29510 raft_consensus.cc:1261] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 1 FOLLOWER]: Refusing update from remote peer 3f004e2132174053b719d6d846b63742: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:28.813445 30012 consensus_queue.cc:1035] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a4c3fa2ac0bc4d9d83586422c50c174a" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:28.813925 30009 consensus_queue.cc:1035] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [LEADER]: Connected to new peer: Peer: permanent_uuid: "498361e242f042cf8297d1f83aa2c413" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:28.846549 29304 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:28.846892 29304 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59502:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:28.850625 29304 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7a78a30a0ba94dc1bc5378c30e392941: Sending DeleteTablet for 3 replicas of tablet b66431ea1b6b43bf871842dfb33f34f7
I20260810 11:42:28.852118 29481 tablet_service.cc:1514] Processing DeleteTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:28 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:56988
I20260810 11:42:28.852332 29876 tablet_service.cc:1514] Processing DeleteTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:28 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46248
I20260810 11:42:28.852658 29681 tablet_service.cc:1514] Processing DeleteTablet for tablet b66431ea1b6b43bf871842dfb33f34f7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:28 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:43026
I20260810 11:42:28.855983 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 29386
I20260810 11:42:28.860291 30036 tablet_replica.cc:330] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: stopping tablet replica
I20260810 11:42:28.860723 30036 raft_consensus.cc:2229] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:28.861527 30036 raft_consensus.cc:2258] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:28.863981 30036 ts_tablet_manager.cc:1905] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:28.864717 30035 tablet_replica.cc:330] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: stopping tablet replica
I20260810 11:42:28.865250 30035 raft_consensus.cc:2229] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:28.865859 30035 raft_consensus.cc:2258] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:28.868525 30034 tablet_replica.cc:330] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: stopping tablet replica
I20260810 11:42:28.870613 30035 ts_tablet_manager.cc:1905] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:28.873010 30036 ts_tablet_manager.cc:1918] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:28.873210 30036 log.cc:1196] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/b66431ea1b6b43bf871842dfb33f34f7
I20260810 11:42:28.873835 30036 ts_tablet_manager.cc:1939] T b66431ea1b6b43bf871842dfb33f34f7 P 3f004e2132174053b719d6d846b63742: Deleting consensus metadata
I20260810 11:42:28.875787 29287 catalog_manager.cc:4841] TS 3f004e2132174053b719d6d846b63742 (127.3.240.194:44833): tablet b66431ea1b6b43bf871842dfb33f34f7 (table test-table [id=85bdef50ba1440338f53094e7398c9a6]) successfully deleted
I20260810 11:42:28.876255 30034 raft_consensus.cc:2229] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:28.876842 30034 raft_consensus.cc:2258] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413 [term 1 FOLLOWER]: Raft consensus is shut down!
Aug 10 11:42:28 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:42:28.877969 30035 ts_tablet_manager.cc:1918] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:28.878140 30035 log.cc:1196] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/b66431ea1b6b43bf871842dfb33f34f7
I20260810 11:42:28.878702 30035 ts_tablet_manager.cc:1939] T b66431ea1b6b43bf871842dfb33f34f7 P a4c3fa2ac0bc4d9d83586422c50c174a: Deleting consensus metadata
I20260810 11:42:28.880501 29287 catalog_manager.cc:4841] TS a4c3fa2ac0bc4d9d83586422c50c174a (127.3.240.195:34957): tablet b66431ea1b6b43bf871842dfb33f34f7 (table test-table [id=85bdef50ba1440338f53094e7398c9a6]) successfully deleted
I20260810 11:42:28.880652 30034 ts_tablet_manager.cc:1905] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:28.886456 30034 ts_tablet_manager.cc:1918] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:28.886650 30034 log.cc:1196] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/b66431ea1b6b43bf871842dfb33f34f7
I20260810 11:42:28.887281 30034 ts_tablet_manager.cc:1939] T b66431ea1b6b43bf871842dfb33f34f7 P 498361e242f042cf8297d1f83aa2c413: Deleting consensus metadata
I20260810 11:42:28.888829 29287 catalog_manager.cc:4841] TS 498361e242f042cf8297d1f83aa2c413 (127.3.240.193:36961): tablet b66431ea1b6b43bf871842dfb33f34f7 (table test-table [id=85bdef50ba1440338f53094e7398c9a6]) successfully deleted
I20260810 11:42:28.891011 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:28.857435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56994 (local address 127.3.240.193:36961)
0810 11:42:28.857621 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:28.857627 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:28.865593 (+ 7966us) server_negotiation.cc:407] Connection header received
0810 11:42:28.865774 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:28.865779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:28.865889 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:28.866008 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:28.867971 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.869196 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.871381 (+ 2185us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:28.872104 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:28.872211 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:28.878682 (+ 6471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:28.878698 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:28.878717 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:28.878766 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:28.887023 (+ 8257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.888064 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.888073 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.888079 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.888258 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:28.888782 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:28.888788 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:28.888792 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:28.889338 (+ 546us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:28.889712 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:28.890338 (+ 626us) server_negotiation.cc:299] Negotiation successful
0810 11:42:28.890582 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:42:28.892043 29570 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:29.054697 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 29386
I20260810 11:42:29.081280 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 29584
Aug 10 11:42:29 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:42:29.102540 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:29.082536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.3.240.194:44833)
0810 11:42:29.082728 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:29.082733 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:29.082752 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:42:29.083025 (+ 273us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:29.083030 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:29.083100 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:29.083201 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:29.085075 (+ 1874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:29.085979 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:29.087963 (+ 1984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:29.088414 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:29.088599 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:29.094134 (+ 5535us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:29.094153 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:29.094176 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:29.094229 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:29.098570 (+ 4341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:29.099515 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:29.099526 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:29.099530 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:29.099713 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:29.100141 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:29.100147 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:29.100151 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:29.100883 (+ 732us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:29.101471 (+ 588us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:29.101700 (+ 229us) server_negotiation.cc:299] Negotiation successful
0810 11:42:29.102036 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 11:42:29.103953 29770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:29.225051 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 29584
I20260810 11:42:29.252542 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 29785
Aug 10 11:42:29 dist-test-slave-drl0 krb5kdc[29243](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362145, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:42:29.271582 30041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:29.253777 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46254 (local address 127.3.240.195:34957)
0810 11:42:29.254312 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:29.254317 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:29.254331 (+ 14us) server_negotiation.cc:407] Connection header received
0810 11:42:29.254388 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:29.254391 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:29.254452 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:29.254576 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:29.256405 (+ 1829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:29.257271 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:29.258734 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:29.259254 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:29.259411 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:29.264537 (+ 5126us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:29.264554 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:29.264572 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:29.264612 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:29.267603 (+ 2991us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:29.268737 (+ 1134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:29.268749 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:29.268752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:29.268973 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:29.269496 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:29.269501 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:29.269504 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:29.269989 (+ 485us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:29.270358 (+ 369us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:29.270718 (+ 360us) server_negotiation.cc:299] Negotiation successful
0810 11:42:29.271102 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":264,"threads_started":1}
I20260810 11:42:29.272434 29967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:29.367424 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 29785
I20260810 11:42:29.395247 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 29261
I20260810 11:42:29.396616 29345 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:29.518244 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 29261
2026-08-10T11:42:29Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5253 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-10T11:42:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:29Z Disabled control of system clock
I20260810 11:42:29.640296 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46201
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:46201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:30.030179 30055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:30.030488 30055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:30.030577 30055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:30.037861 30055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:30.037966 30055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:30.038040 30055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:30.038128 30055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:30.038189 30055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:30.048218 30055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:46201
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:46201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:30.049630 30055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:30.051568 30055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:30.064173 30068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:30.064335 30065 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:30.064373 30066 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:30.065717 30055 server_base.cc:1034] running on GCE node
I20260810 11:42:30.066992 30055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:30.068944 30055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:30.070225 30055 hybrid_clock.cc:648] HybridClock initialized: now 1786362150070179 us; error 31 us; skew 500 ppm
I20260810 11:42:30.070803 30055 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:30.073681 30055 webserver.cc:460] Webserver started at http://127.3.240.254:33335/ using document root <none> and password file <none>
I20260810 11:42:30.074455 30055 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:30.074551 30055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:30.074915 30055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:30.078661 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "f9d20198318d4080aa233b2366c7b4cc"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "7d9424637fcfaed4933f31ad98021446"
server_key_iv: "0247531a96258a2657357cd9f77220a4"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.079895 30055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "f9d20198318d4080aa233b2366c7b4cc"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "7d9424637fcfaed4933f31ad98021446"
server_key_iv: "0247531a96258a2657357cd9f77220a4"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.085858 30055 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:42:30.090150 30074 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:30.092046 30055 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:42:30.092262 30055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "f9d20198318d4080aa233b2366c7b4cc"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "7d9424637fcfaed4933f31ad98021446"
server_key_iv: "0247531a96258a2657357cd9f77220a4"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.092490 30055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:30.108281 30055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:30.109155 30055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:30.109462 30055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:30.131973 30055 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:46201
I20260810 11:42:30.131999 30149 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:46201 every 8 connection(s)
I20260810 11:42:30.133370 30055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:30.137260 30150 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:30.138767 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30055
I20260810 11:42:30.138994 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:30.139634 4035 external_mini_cluster.cc:1428] Setting key 57be0e4955e584feb9151b87b2283e6c
I20260810 11:42:30.150234 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: Bootstrap starting.
I20260810 11:42:30.154867 30150 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:30.156709 30150 log.cc:824] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:30.157411 30155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:30.142254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36764 (local address 127.3.240.254:46201)
0810 11:42:30.143742 (+ 1488us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:30.143763 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:30.143827 (+ 64us) server_negotiation.cc:407] Connection header received
0810 11:42:30.146152 (+ 2325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:30.146186 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:30.146859 (+ 673us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:30.147428 (+ 569us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:30.148147 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.149486 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.151468 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.152615 (+ 1147us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.152765 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:30.154442 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:30.154450 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:30.154465 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:30.155238 (+ 773us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:30.155357 (+ 119us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:30.155381 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:30.155504 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:30.155870 (+ 366us) server_negotiation.cc:299] Negotiation successful
0810 11:42:30.156248 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":947,"thread_start_us":616,"threads_started":1}
I20260810 11:42:30.161387 30150 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: No bootstrap required, opened a new log
I20260810 11:42:30.169024 30150 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } }
I20260810 11:42:30.169394 30150 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:30.169507 30150 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9d20198318d4080aa233b2366c7b4cc, State: Initialized, Role: FOLLOWER
I20260810 11:42:30.170156 30150 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } }
I20260810 11:42:30.170478 30150 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:30.170634 30150 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:30.170818 30150 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:30.172969 30150 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } }
I20260810 11:42:30.173496 30150 leader_election.cc:302] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9d20198318d4080aa233b2366c7b4cc; no voters:
I20260810 11:42:30.174153 30150 leader_election.cc:288] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:30.174268 30157 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:30.175105 30157 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [term 1 LEADER]: Becoming Leader. State: Replica: f9d20198318d4080aa233b2366c7b4cc, State: Running, Role: LEADER
I20260810 11:42:30.175719 30157 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } }
I20260810 11:42:30.177891 30150 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:30.178877 30161 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9d20198318d4080aa233b2366c7b4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } } }
I20260810 11:42:30.178905 30158 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9d20198318d4080aa233b2366c7b4cc. Latest consensus state: current_term: 1 leader_uuid: "f9d20198318d4080aa233b2366c7b4cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9d20198318d4080aa233b2366c7b4cc" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 46201 } } }
I20260810 11:42:30.179425 30161 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:30.179463 30158 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:30.182173 30164 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:30.187983 30164 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:30.202183 30164 catalog_manager.cc:1347] Generated new cluster ID: 4977f26cdee240d8a6679fc4f42f7f03
I20260810 11:42:30.202310 30164 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:30.225478 30164 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:30.226759 30164 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:30.241237 30164 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: Generated new TSK 0
I20260810 11:42:30.242111 30164 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:30.253696 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--builtin_ntp_servers=127.3.240.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:30.645100 30182 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:30.645439 30182 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:30.645608 30182 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:30.652827 30182 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:30.652973 30182 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:30.653153 30182 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:30.662981 30182 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:30.664429 30182 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:30.666222 30182 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:30.680799 30193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:30.680891 30196 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:30.681115 30191 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:30.682478 30182 server_base.cc:1034] running on GCE node
I20260810 11:42:30.683195 30182 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:30.684224 30182 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:30.685460 30182 hybrid_clock.cc:648] HybridClock initialized: now 1786362150685436 us; error 51 us; skew 500 ppm
I20260810 11:42:30.685828 30182 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:30.688730 30182 webserver.cc:460] Webserver started at http://127.3.240.193:33779/ using document root <none> and password file <none>
I20260810 11:42:30.689463 30182 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:30.689559 30182 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:30.689886 30182 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:30.692795 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "d958d46122ee4be4a6382952cc44a6c6"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "a1e46e80d925c72b879277185dfcaaee"
server_key_iv: "cfc219203b58d65d3492d66cbb767528"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.693926 30182 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "d958d46122ee4be4a6382952cc44a6c6"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "a1e46e80d925c72b879277185dfcaaee"
server_key_iv: "cfc219203b58d65d3492d66cbb767528"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.700964 30182 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260810 11:42:30.704965 30204 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:30.706295 30182 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260810 11:42:30.706493 30182 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "d958d46122ee4be4a6382952cc44a6c6"
format_stamp: "Formatted at 2026-08-10 11:42:30 on dist-test-slave-drl0"
server_key: "a1e46e80d925c72b879277185dfcaaee"
server_key_iv: "cfc219203b58d65d3492d66cbb767528"
server_key_version: "encryptionkey@0"
I20260810 11:42:30.706713 30182 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:30.726325 30182 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:30.727222 30182 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:30.727557 30182 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:30.728731 30182 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:30.730329 30182 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:30.730428 30182 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:30.730564 30182 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:30.730684 30182 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:30.773370 30182 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:37901
I20260810 11:42:30.773406 30359 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:37901 every 8 connection(s)
I20260810 11:42:30.774739 30182 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:30.775907 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30182
I20260810 11:42:30.776100 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:30.776739 4035 external_mini_cluster.cc:1428] Setting key 8bce44aaf30fed01adb85d3277d680c4
I20260810 11:42:30.780527 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--builtin_ntp_servers=127.3.240.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:30.792294 30363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:30.778188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:48291 (local address 127.3.240.254:46201)
0810 11:42:30.778919 (+ 731us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:30.778960 (+ 41us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:30.780915 (+ 1955us) server_negotiation.cc:407] Connection header received
0810 11:42:30.783462 (+ 2547us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:30.783470 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:30.783646 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:30.783766 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:30.785603 (+ 1837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.786368 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.789056 (+ 2688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.789466 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.789580 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:30.791433 (+ 1853us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:30.791438 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:30.791441 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:30.791470 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:30.791515 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:30.791527 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:30.791721 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:30.791942 (+ 221us) server_negotiation.cc:299] Negotiation successful
0810 11:42:30.792027 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":422,"threads_started":1}
I20260810 11:42:30.793687 30364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:30.779245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46201 (local address 127.3.240.193:48291)
0810 11:42:30.780694 (+ 1449us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:30.780763 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:30.783032 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:30.784250 (+ 1218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:30.784264 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:30.784398 (+ 134us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:30.785385 (+ 987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:30.785415 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.786550 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.786558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:30.788878 (+ 2320us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:30.788898 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:30.789716 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:30.789725 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:30.790068 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:30.791029 (+ 961us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:30.791034 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:30.791093 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:30.791211 (+ 118us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:30.791632 (+ 421us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:30.791646 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:30.791656 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 11:42:30.791847 (+ 191us) client_negotiation.cc:240] Negotiation successful
0810 11:42:30.792295 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":761,"thread_start_us":480,"threads_started":1}
I20260810 11:42:30.796032 30361 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46201
I20260810 11:42:30.796464 30361 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:30.797327 30361 heartbeater.cc:511] Master 127.3.240.254:46201 requested a full tablet report, sending...
I20260810 11:42:30.800308 30096 ts_manager.cc:194] Registered new tserver with Master: d958d46122ee4be4a6382952cc44a6c6 (127.3.240.193:37901)
I20260810 11:42:30.803087 30096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:48291
W20260810 11:42:31.177380 30366 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:31.178102 30366 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:31.178398 30366 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:31.186465 30366 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:31.186561 30366 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:31.186668 30366 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:31.198272 30366 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:31.200068 30366 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:31.202208 30366 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:31.217067 30376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:31.217127 30379 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:31.217165 30375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:31.218046 30366 server_base.cc:1034] running on GCE node
I20260810 11:42:31.218868 30366 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:31.220340 30366 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:31.221604 30366 hybrid_clock.cc:648] HybridClock initialized: now 1786362151221518 us; error 70 us; skew 500 ppm
I20260810 11:42:31.222085 30366 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:31.224798 30366 webserver.cc:460] Webserver started at http://127.3.240.194:46123/ using document root <none> and password file <none>
I20260810 11:42:31.225492 30366 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:31.225584 30366 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:31.225869 30366 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:31.228741 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "c7b09b0813494ca189dd67ef3d606738"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "cb19b7d8ebcc727ff712b933510368f0"
server_key_iv: "5ae81c45ac902046f7ee3e2b91d32da2"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.229487 30366 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "c7b09b0813494ca189dd67ef3d606738"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "cb19b7d8ebcc727ff712b933510368f0"
server_key_iv: "5ae81c45ac902046f7ee3e2b91d32da2"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.234629 30366 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 11:42:31.240808 30386 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.244338 30366 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.004s sys 0.001s
I20260810 11:42:31.244526 30366 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "c7b09b0813494ca189dd67ef3d606738"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "cb19b7d8ebcc727ff712b933510368f0"
server_key_iv: "5ae81c45ac902046f7ee3e2b91d32da2"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.244788 30366 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:31.265645 30366 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:31.266448 30366 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:31.266736 30366 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:31.267910 30366 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:31.269409 30366 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:31.269498 30366 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.269569 30366 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:31.269624 30366 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.311933 30366 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:32919
I20260810 11:42:31.311985 30539 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:32919 every 8 connection(s)
I20260810 11:42:31.313300 30366 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:31.314684 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30366
I20260810 11:42:31.315037 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:31.315619 4035 external_mini_cluster.cc:1428] Setting key e1339df2c1e65855dd3893197b2942da
I20260810 11:42:31.319433 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--builtin_ntp_servers=127.3.240.212:34339
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:31.328574 30547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.316154 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:42855 (local address 127.3.240.254:46201)
0810 11:42:31.316667 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.316674 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.318232 (+ 1558us) server_negotiation.cc:407] Connection header received
0810 11:42:31.320752 (+ 2520us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.320758 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.320856 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.321000 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:31.322473 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.323805 (+ 1332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.325595 (+ 1790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.326046 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.326190 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.327688 (+ 1498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:31.327694 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:31.327699 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:31.327726 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:31.327767 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:31.327779 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:31.327890 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.328115 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.328218 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":380,"thread_start_us":273,"threads_started":1}
I20260810 11:42:31.329681 30548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.316857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46201 (local address 127.3.240.194:42855)
0810 11:42:31.318075 (+ 1218us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:31.318124 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:31.320447 (+ 2323us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:31.321222 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:31.321230 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:31.321323 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:31.322153 (+ 830us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.322191 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.324018 (+ 1827us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.324024 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.325418 (+ 1394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.325434 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.326281 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.326288 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.326534 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.327356 (+ 822us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:31.327359 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:31.327406 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:31.327487 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:31.327879 (+ 392us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:31.327890 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:31.327900 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 11:42:31.328104 (+ 204us) client_negotiation.cc:240] Negotiation successful
0810 11:42:31.328508 (+ 404us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":317,"threads_started":1}
I20260810 11:42:31.331346 30542 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46201
I20260810 11:42:31.331667 30542 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:31.332440 30542 heartbeater.cc:511] Master 127.3.240.254:46201 requested a full tablet report, sending...
I20260810 11:42:31.333865 30096 ts_manager.cc:194] Registered new tserver with Master: c7b09b0813494ca189dd67ef3d606738 (127.3.240.194:32919)
I20260810 11:42:31.334789 30096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:42855
W20260810 11:42:31.693849 30549 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:31.694172 30549 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:31.694343 30549 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:31.701839 30549 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:31.701975 30549 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:31.702149 30549 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:31.711843 30549 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34339
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:46201
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:31.713174 30549 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:31.715211 30549 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:31.729146 30557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:31.729295 30558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:31.729271 30560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:31.730095 30549 server_base.cc:1034] running on GCE node
I20260810 11:42:31.730710 30549 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:31.731866 30549 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:31.733119 30549 hybrid_clock.cc:648] HybridClock initialized: now 1786362151733072 us; error 39 us; skew 500 ppm
I20260810 11:42:31.733474 30549 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:31.736002 30549 webserver.cc:460] Webserver started at http://127.3.240.195:41373/ using document root <none> and password file <none>
I20260810 11:42:31.736711 30549 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:31.736846 30549 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:31.737179 30549 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:31.740116 30549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "5e9fae79df134773a442faa19e1d5230"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "303052cc24cb3ebdaa18e56ae21cce5a"
server_key_iv: "28cf1ae5b0956f71b9c6360a5cad4d78"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.740900 30549 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "5e9fae79df134773a442faa19e1d5230"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "303052cc24cb3ebdaa18e56ae21cce5a"
server_key_iv: "28cf1ae5b0956f71b9c6360a5cad4d78"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.746208 30549 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:42:31.750367 30572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.752085 30549 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:42:31.752282 30549 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "5e9fae79df134773a442faa19e1d5230"
format_stamp: "Formatted at 2026-08-10 11:42:31 on dist-test-slave-drl0"
server_key: "303052cc24cb3ebdaa18e56ae21cce5a"
server_key_iv: "28cf1ae5b0956f71b9c6360a5cad4d78"
server_key_version: "encryptionkey@0"
I20260810 11:42:31.752530 30549 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:31.783931 30549 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:31.784888 30549 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:31.785197 30549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:31.786208 30549 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:31.788286 30549 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:31.788439 30549 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.788582 30549 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:31.788678 30549 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:31.807096 30361 heartbeater.cc:503] Master 127.3.240.254:46201 was elected leader, sending a full tablet report...
I20260810 11:42:31.831284 30549 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:36883
I20260810 11:42:31.831315 30733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:36883 every 8 connection(s)
I20260810 11:42:31.832693 30549 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:31.835145 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30549
I20260810 11:42:31.835340 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:31.835953 4035 external_mini_cluster.cc:1428] Setting key 1a1a78e60ee114978032cf40c836e470
I20260810 11:42:31.846724 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.835746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44803 (local address 127.3.240.254:46201)
0810 11:42:31.836239 (+ 493us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.836243 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.837647 (+ 1404us) server_negotiation.cc:407] Connection header received
0810 11:42:31.839748 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.839752 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.839817 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.839915 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:31.840981 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.841742 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.843529 (+ 1787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.843919 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.844017 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.845826 (+ 1809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:31.845831 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:31.845835 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:31.845853 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:31.845897 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:31.845909 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:31.845995 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.846327 (+ 332us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.846417 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":298,"threads_started":1}
I20260810 11:42:31.848237 30739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.836352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:46201 (local address 127.3.240.195:44803)
0810 11:42:31.837477 (+ 1125us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:31.837522 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:31.839493 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:31.840143 (+ 650us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:31.840150 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:31.840231 (+ 81us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:31.840790 (+ 559us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.840810 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.841941 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.841945 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.843334 (+ 1389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.843347 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.844138 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.844144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.844339 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.845310 (+ 971us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:31.845314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:31.845390 (+ 76us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:31.845518 (+ 128us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:31.846065 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:31.846076 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:31.846083 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 11:42:31.846264 (+ 181us) client_negotiation.cc:240] Negotiation successful
0810 11:42:31.846555 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":376,"threads_started":1}
I20260810 11:42:31.850075 30734 heartbeater.cc:347] Connected to a master server at 127.3.240.254:46201
I20260810 11:42:31.850430 30734 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:31.851315 30734 heartbeater.cc:511] Master 127.3.240.254:46201 requested a full tablet report, sending...
I20260810 11:42:31.852807 30096 ts_manager.cc:194] Registered new tserver with Master: 5e9fae79df134773a442faa19e1d5230 (127.3.240.195:36883)
I20260810 11:42:31.853627 30096 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:44803
I20260810 11:42:31.864128 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:31.876533 30736 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.869137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36778 (local address 127.3.240.254:46201)
0810 11:42:31.869380 (+ 243us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.869385 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.870057 (+ 672us) server_negotiation.cc:407] Connection header received
0810 11:42:31.870115 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.870119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.870196 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.870377 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:31.871280 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.872220 (+ 940us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.873765 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.874184 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.874297 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.875480 (+ 1183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:31.875485 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:31.875488 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:31.875505 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:31.875536 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:31.875547 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:31.875621 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.876030 (+ 409us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.876172 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260810 11:42:31.880910 30096 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:36778:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:31.888113 30096 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:31.926568 30754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.902878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34002)
0810 11:42:31.915790 (+ 12912us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:31.915854 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:31.916068 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:31.917777 (+ 1709us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:31.917789 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:31.917835 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:31.918348 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.918372 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.920577 (+ 2205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.920582 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.923187 (+ 2605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.923202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.925560 (+ 2358us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.925567 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.925773 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.925828 (+ 55us) client_negotiation.cc:769] Sending connection context
0810 11:42:31.925961 (+ 133us) client_negotiation.cc:240] Negotiation successful
0810 11:42:31.926156 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12510,"mutex_wait_us":40,"thread_start_us":12255,"threads_started":1}
I20260810 11:42:31.927309 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.903131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34002 (local address 127.3.240.193:37901)
0810 11:42:31.916099 (+ 12968us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.916111 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.916146 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:42:31.916634 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.916647 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.916990 (+ 343us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.917215 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:31.918664 (+ 1449us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.920389 (+ 1725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.923469 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.924984 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.925113 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.925403 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.926436 (+ 1033us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.926761 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12779,"thread_start_us":13146,"threads_started":1}
I20260810 11:42:31.928345 30755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.902880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:32919 (local address 127.0.0.1:35864)
0810 11:42:31.917176 (+ 14296us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:31.917195 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:31.917319 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:31.920330 (+ 3011us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:31.920335 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:31.920363 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:31.920870 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.920882 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.923094 (+ 2212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.923099 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.925349 (+ 2250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.925368 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.927304 (+ 1936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.927314 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.927531 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.927760 (+ 229us) client_negotiation.cc:769] Sending connection context
0810 11:42:31.927839 (+ 79us) client_negotiation.cc:240] Negotiation successful
0810 11:42:31.927984 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14169,"mutex_wait_us":121,"thread_start_us":14081,"threads_started":1}
I20260810 11:42:31.929455 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.903087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35864 (local address 127.3.240.194:32919)
0810 11:42:31.919591 (+ 16504us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.919600 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.919629 (+ 29us) server_negotiation.cc:407] Connection header received
0810 11:42:31.919715 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.919722 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.919950 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.921117 (+ 1167us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:31.921336 (+ 219us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.922887 (+ 1551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.925590 (+ 2703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.927068 (+ 1478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.928576 (+ 1508us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.928816 (+ 240us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.928910 (+ 94us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.929084 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16336,"thread_start_us":16994,"threads_started":1}
I20260810 11:42:31.931659 30264 tablet_service.cc:1467] Processing CreateTablet for tablet 522775451bbe4212b67bfa7299536cb0 (DEFAULT_TABLE table=test-table [id=4da7c0283dc9499fabc602656dd0addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:31.933151 30264 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522775451bbe4212b67bfa7299536cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:31.933229 30458 tablet_service.cc:1467] Processing CreateTablet for tablet 522775451bbe4212b67bfa7299536cb0 (DEFAULT_TABLE table=test-table [id=4da7c0283dc9499fabc602656dd0addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:31.934427 30458 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522775451bbe4212b67bfa7299536cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:31.937223 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.917722 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36883 (local address 127.0.0.1:55166)
0810 11:42:31.925898 (+ 8176us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:31.925921 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:31.926061 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:31.929914 (+ 3853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:31.929919 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:31.929946 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:31.930257 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.930267 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.932372 (+ 2105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.932377 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.934561 (+ 2184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:31.934576 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.936449 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.936455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:31.936633 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.936667 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 11:42:31.936783 (+ 116us) client_negotiation.cc:240] Negotiation successful
0810 11:42:31.936917 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8005,"thread_start_us":8443,"threads_started":1}
I20260810 11:42:31.937758 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:31.917828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55166 (local address 127.3.240.195:36883)
0810 11:42:31.929304 (+ 11476us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:31.929321 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:31.929365 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:42:31.929476 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:31.929485 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:31.929721 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:31.929922 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:31.930437 (+ 515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.932184 (+ 1747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.934773 (+ 2589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:31.936247 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:31.936405 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:31.936693 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:31.936831 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 11:42:31.937004 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11291,"thread_start_us":11120,"threads_started":1}
I20260810 11:42:31.941507 30639 tablet_service.cc:1467] Processing CreateTablet for tablet 522775451bbe4212b67bfa7299536cb0 (DEFAULT_TABLE table=test-table [id=4da7c0283dc9499fabc602656dd0addf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:31.943121 30639 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 522775451bbe4212b67bfa7299536cb0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:31.948128 30763 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Bootstrap starting.
I20260810 11:42:31.949961 30764 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Bootstrap starting.
I20260810 11:42:31.952441 30763 tablet_bootstrap.cc:654] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:31.953575 30764 tablet_bootstrap.cc:654] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:31.953812 30763 log.cc:824] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:31.954787 30764 log.cc:824] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:31.957387 30763 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: No bootstrap required, opened a new log
I20260810 11:42:31.957749 30763 ts_tablet_manager.cc:1397] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.006s
I20260810 11:42:31.957954 30764 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: No bootstrap required, opened a new log
I20260810 11:42:31.958220 30764 ts_tablet_manager.cc:1397] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260810 11:42:31.959133 30769 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Bootstrap starting.
I20260810 11:42:31.962779 30769 tablet_bootstrap.cc:654] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:31.964046 30769 log.cc:824] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:31.966492 30764 raft_consensus.cc:348] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.966912 30764 raft_consensus.cc:374] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:31.967039 30764 raft_consensus.cc:729] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d958d46122ee4be4a6382952cc44a6c6, State: Initialized, Role: FOLLOWER
I20260810 11:42:31.967671 30764 consensus_queue.cc:260] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.968166 30769 tablet_bootstrap.cc:492] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: No bootstrap required, opened a new log
I20260810 11:42:31.968461 30769 ts_tablet_manager.cc:1397] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.004s
I20260810 11:42:31.968462 30763 raft_consensus.cc:348] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.968909 30763 raft_consensus.cc:374] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:31.969007 30763 raft_consensus.cc:729] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c7b09b0813494ca189dd67ef3d606738, State: Initialized, Role: FOLLOWER
I20260810 11:42:31.969650 30764 ts_tablet_manager.cc:1428] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Time spent starting tablet: real 0.011s user 0.013s sys 0.000s
I20260810 11:42:31.969581 30763 consensus_queue.cc:260] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.970772 30542 heartbeater.cc:503] Master 127.3.240.254:46201 was elected leader, sending a full tablet report...
I20260810 11:42:31.971342 30763 ts_tablet_manager.cc:1428] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Time spent starting tablet: real 0.013s user 0.013s sys 0.001s
I20260810 11:42:31.978055 30769 raft_consensus.cc:348] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.978473 30769 raft_consensus.cc:374] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:31.978551 30769 raft_consensus.cc:729] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5e9fae79df134773a442faa19e1d5230, State: Initialized, Role: FOLLOWER
I20260810 11:42:31.979230 30769 consensus_queue.cc:260] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:31.980389 30734 heartbeater.cc:503] Master 127.3.240.254:46201 was elected leader, sending a full tablet report...
I20260810 11:42:31.981089 30769 ts_tablet_manager.cc:1428] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Time spent starting tablet: real 0.012s user 0.011s sys 0.001s
I20260810 11:42:32.011246 30771 raft_consensus.cc:482] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:32.011819 30771 raft_consensus.cc:504] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:32.014551 30771 leader_election.cc:288] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c7b09b0813494ca189dd67ef3d606738 (127.3.240.194:32919), 5e9fae79df134773a442faa19e1d5230 (127.3.240.195:36883)
I20260810 11:42:32.023451 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.015129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:54043 (local address 127.3.240.194:32919)
0810 11:42:32.015376 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.015385 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.016055 (+ 670us) server_negotiation.cc:407] Connection header received
0810 11:42:32.016337 (+ 282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.016345 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.016437 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.016506 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:32.017477 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.018400 (+ 923us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.021248 (+ 2848us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.022441 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.022610 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.022687 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.023094 (+ 407us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.023202 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:42:32.023706 30775 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.015162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:32919 (local address 127.3.240.193:54043)
0810 11:42:32.015915 (+ 753us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.015939 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:32.016136 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:32.016735 (+ 599us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:32.016743 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:32.016778 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:32.017304 (+ 526us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:32.017321 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.018569 (+ 1248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.018573 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:32.021003 (+ 2430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:32.021024 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.022657 (+ 1633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.022662 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:32.022778 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.022803 (+ 25us) client_negotiation.cc:769] Sending connection context
0810 11:42:32.023069 (+ 266us) client_negotiation.cc:240] Negotiation successful
0810 11:42:32.023213 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":598,"mutex_wait_us":67,"thread_start_us":419,"threads_started":1}
I20260810 11:42:32.023890 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.016449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:50369 (local address 127.3.240.195:36883)
0810 11:42:32.016656 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.016661 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.017360 (+ 699us) server_negotiation.cc:407] Connection header received
0810 11:42:32.017548 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.017554 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.017653 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.017757 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:32.018318 (+ 561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.019133 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.021537 (+ 2404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.022800 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.022973 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.023074 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.023478 (+ 404us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.023615 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:42:32.024098 30776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.016131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:36883 (local address 127.3.240.193:50369)
0810 11:42:32.017242 (+ 1111us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.017263 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:32.017398 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:32.017809 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:32.017812 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:32.017832 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:32.018159 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:32.018167 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.019280 (+ 1113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.019286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:32.021357 (+ 2071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:32.021367 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.023092 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.023098 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:32.023276 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.023309 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:42:32.023399 (+ 90us) client_negotiation.cc:240] Negotiation successful
0810 11:42:32.023515 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":962,"mutex_wait_us":45,"thread_start_us":743,"threads_started":1}
I20260810 11:42:32.024895 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522775451bbe4212b67bfa7299536cb0" candidate_uuid: "d958d46122ee4be4a6382952cc44a6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b09b0813494ca189dd67ef3d606738" is_pre_election: true
I20260810 11:42:32.025056 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522775451bbe4212b67bfa7299536cb0" candidate_uuid: "d958d46122ee4be4a6382952cc44a6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9fae79df134773a442faa19e1d5230" is_pre_election: true
I20260810 11:42:32.025473 30480 raft_consensus.cc:2454] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d958d46122ee4be4a6382952cc44a6c6 in term 0.
I20260810 11:42:32.025573 30665 raft_consensus.cc:2454] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d958d46122ee4be4a6382952cc44a6c6 in term 0.
I20260810 11:42:32.026208 30208 leader_election.cc:302] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c7b09b0813494ca189dd67ef3d606738, d958d46122ee4be4a6382952cc44a6c6; no voters:
I20260810 11:42:32.026604 30771 raft_consensus.cc:2781] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:32.026782 30771 raft_consensus.cc:482] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:32.026906 30771 raft_consensus.cc:3037] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 0 FOLLOWER]: Advancing to term 1
W20260810 11:42:32.027745 30362 tablet.cc:2369] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:32.030028 30771 raft_consensus.cc:504] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:32.031319 30771 leader_election.cc:288] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [CANDIDATE]: Term 1 election: Requested vote from peers c7b09b0813494ca189dd67ef3d606738 (127.3.240.194:32919), 5e9fae79df134773a442faa19e1d5230 (127.3.240.195:36883)
I20260810 11:42:32.031904 30480 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522775451bbe4212b67bfa7299536cb0" candidate_uuid: "d958d46122ee4be4a6382952cc44a6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c7b09b0813494ca189dd67ef3d606738"
I20260810 11:42:32.032155 30480 raft_consensus.cc:3037] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:32.032150 30665 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "522775451bbe4212b67bfa7299536cb0" candidate_uuid: "d958d46122ee4be4a6382952cc44a6c6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5e9fae79df134773a442faa19e1d5230"
I20260810 11:42:32.032480 30665 raft_consensus.cc:3037] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:32.035123 30665 raft_consensus.cc:2454] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d958d46122ee4be4a6382952cc44a6c6 in term 1.
I20260810 11:42:32.035621 30208 leader_election.cc:302] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5e9fae79df134773a442faa19e1d5230, d958d46122ee4be4a6382952cc44a6c6; no voters:
I20260810 11:42:32.035774 30480 raft_consensus.cc:2454] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d958d46122ee4be4a6382952cc44a6c6 in term 1.
I20260810 11:42:32.035974 30771 raft_consensus.cc:2781] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:32.036536 30771 raft_consensus.cc:686] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 1 LEADER]: Becoming Leader. State: Replica: d958d46122ee4be4a6382952cc44a6c6, State: Running, Role: LEADER
I20260810 11:42:32.037138 30771 consensus_queue.cc:237] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } }
I20260810 11:42:32.042452 30093 catalog_manager.cc:5495] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 reported cstate change: term changed from 0 to 1, leader changed from <none> to d958d46122ee4be4a6382952cc44a6c6 (127.3.240.193). New cstate: current_term: 1 leader_uuid: "d958d46122ee4be4a6382952cc44a6c6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d958d46122ee4be4a6382952cc44a6c6" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 37901 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 } health_report { overall_health: UNKNOWN } } }
W20260810 11:42:32.066766 30545 tablet.cc:2369] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:42:32.084642 30735 tablet.cc:2369] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:32.086437 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.079936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34018 (local address 127.3.240.193:37901)
0810 11:42:32.080128 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.080132 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.080152 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:42:32.080267 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.080271 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.080380 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.080525 (+ 145us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:32.081047 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.081738 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.083539 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.084111 (+ 572us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.084452 (+ 341us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.085286 (+ 834us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:32.085811 (+ 525us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:32.085988 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.086074 (+ 86us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.086170 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:42:32.094875 30480 raft_consensus.cc:1261] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 1 FOLLOWER]: Refusing update from remote peer d958d46122ee4be4a6382952cc44a6c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:32.094862 30665 raft_consensus.cc:1261] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 1 FOLLOWER]: Refusing update from remote peer d958d46122ee4be4a6382952cc44a6c6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:32.096004 30781 consensus_queue.cc:1035] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c7b09b0813494ca189dd67ef3d606738" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 32919 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:32.096491 30771 consensus_queue.cc:1035] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5e9fae79df134773a442faa19e1d5230" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 36883 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:32.108343 30786 mvcc.cc:205] Tried to move back new op lower bound from 7316939374969434112 to 7316939374748532736. Current Snapshot: MvccSnapshot[applied={T|T < 7316939374969434112}]
I20260810 11:42:32.110203 30791 mvcc.cc:205] Tried to move back new op lower bound from 7316939374969434112 to 7316939374748532736. Current Snapshot: MvccSnapshot[applied={T|T < 7316939374969434112}]
I20260810 11:42:32.112411 30793 mvcc.cc:205] Tried to move back new op lower bound from 7316939374969434112 to 7316939374748532736. Current Snapshot: MvccSnapshot[applied={T|T < 7316939374969434112}]
I20260810 11:42:32.125949 30093 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:36778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:32.126246 30093 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:36778:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:32.130087 30093 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9d20198318d4080aa233b2366c7b4cc: Sending DeleteTablet for 3 replicas of tablet 522775451bbe4212b67bfa7299536cb0
I20260810 11:42:32.131500 30458 tablet_service.cc:1514] Processing DeleteTablet for tablet 522775451bbe4212b67bfa7299536cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:32 UTC) from {username='slave'} at 127.0.0.1:35864
I20260810 11:42:32.131508 30264 tablet_service.cc:1514] Processing DeleteTablet for tablet 522775451bbe4212b67bfa7299536cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:32 UTC) from {username='slave'} at 127.0.0.1:34002
I20260810 11:42:32.131812 30639 tablet_service.cc:1514] Processing DeleteTablet for tablet 522775451bbe4212b67bfa7299536cb0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:32 UTC) from {username='slave'} at 127.0.0.1:55166
I20260810 11:42:32.134506 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30182
I20260810 11:42:32.147119 30804 tablet_replica.cc:330] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: stopping tablet replica
I20260810 11:42:32.147615 30804 raft_consensus.cc:2229] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:32.148149 30804 raft_consensus.cc:2258] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:32.151556 30804 ts_tablet_manager.cc:1905] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:32.155511 30803 tablet_replica.cc:330] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: stopping tablet replica
I20260810 11:42:32.156020 30803 raft_consensus.cc:2229] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:32.156967 30802 tablet_replica.cc:330] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: stopping tablet replica
I20260810 11:42:32.157385 30802 raft_consensus.cc:2229] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:32.157871 30802 raft_consensus.cc:2258] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:32.157493 30803 raft_consensus.cc:2258] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:32.160986 30757 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.138598 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34020 (local address 127.3.240.193:37901)
0810 11:42:32.139263 (+ 665us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.139273 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.148931 (+ 9658us) server_negotiation.cc:407] Connection header received
0810 11:42:32.149851 (+ 920us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.149857 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.149974 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.150044 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:32.151575 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.152789 (+ 1214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.156829 (+ 4040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.157559 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.157655 (+ 96us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.159271 (+ 1616us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:32.159283 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:32.159300 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:32.160066 (+ 766us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:32.160122 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:32.160154 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:32.160250 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.160484 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.160596 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260810 11:42:32.162590 30354 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:32.163990 30802 ts_tablet_manager.cc:1905] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:32.165045 30804 ts_tablet_manager.cc:1918] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:32.165337 30804 log.cc:1196] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/522775451bbe4212b67bfa7299536cb0
I20260810 11:42:32.166144 30804 ts_tablet_manager.cc:1939] T 522775451bbe4212b67bfa7299536cb0 P 5e9fae79df134773a442faa19e1d5230: Deleting consensus metadata
I20260810 11:42:32.168912 30077 catalog_manager.cc:4841] TS 5e9fae79df134773a442faa19e1d5230 (127.3.240.195:36883): tablet 522775451bbe4212b67bfa7299536cb0 (table test-table [id=4da7c0283dc9499fabc602656dd0addf]) successfully deleted
I20260810 11:42:32.172530 30802 ts_tablet_manager.cc:1918] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:32.172670 30802 log.cc:1196] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/522775451bbe4212b67bfa7299536cb0
I20260810 11:42:32.173151 30802 ts_tablet_manager.cc:1939] T 522775451bbe4212b67bfa7299536cb0 P c7b09b0813494ca189dd67ef3d606738: Deleting consensus metadata
I20260810 11:42:32.174700 30077 catalog_manager.cc:4841] TS c7b09b0813494ca189dd67ef3d606738 (127.3.240.194:32919): tablet 522775451bbe4212b67bfa7299536cb0 (table test-table [id=4da7c0283dc9499fabc602656dd0addf]) successfully deleted
I20260810 11:42:32.164287 30803 ts_tablet_manager.cc:1905] T 522775451bbe4212b67bfa7299536cb0 P d958d46122ee4be4a6382952cc44a6c6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:32.267098 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30182
W20260810 11:42:32.295903 30079 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:37901 (error 108)
I20260810 11:42:32.296183 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30366
I20260810 11:42:32.298482 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.298031 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34022)
0810 11:42:32.298193 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.298295 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20260810 11:42:32.298738 30079 catalog_manager.cc:4584] TS d958d46122ee4be4a6382952cc44a6c6 (127.3.240.193:37901): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 522775451bbe4212b67bfa7299536cb0: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
I20260810 11:42:32.303681 30756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.297200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35878 (local address 127.3.240.194:32919)
0810 11:42:32.297419 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.297426 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.297448 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:42:32.297716 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.297722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.297859 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.297957 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:32.298455 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.299364 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.300669 (+ 1305us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.301103 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.301211 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.302273 (+ 1062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:32.302281 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:32.302294 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:32.302885 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:32.302996 (+ 111us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:32.303019 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:32.303112 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.303309 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.303418 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260810 11:42:32.304906 30534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:32.352496 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.351984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34030)
0810 11:42:32.352229 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.352324 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108}
I20260810 11:42:32.353780 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.353228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34034)
0810 11:42:32.353407 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.353480 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 11:42:32.405298 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30366
I20260810 11:42:32.415107 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.414367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34044)
0810 11:42:32.414730 (+ 363us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.414865 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20260810 11:42:32.416193 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.415853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34046)
0810 11:42:32.416015 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.416092 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260810 11:42:32.429770 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30549
I20260810 11:42:32.438190 30760 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.430878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55180 (local address 127.3.240.195:36883)
0810 11:42:32.431104 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:32.431110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:32.431133 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:42:32.431327 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:32.431331 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:32.431434 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:32.431565 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:32.431983 (+ 418us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.432777 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.434544 (+ 1767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:32.435237 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:32.435392 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:32.436749 (+ 1357us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:32.436757 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:32.436772 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:32.437347 (+ 575us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:32.437394 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:32.437414 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:32.437557 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:32.437660 (+ 103us) server_negotiation.cc:299] Negotiation successful
0810 11:42:32.437769 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 11:42:32.438848 30725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:32.497076 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.496450 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34054)
0810 11:42:32.496738 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.496864 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
I20260810 11:42:32.498339 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.497812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34060)
0810 11:42:32.498123 (+ 311us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.498215 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260810 11:42:32.536410 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30549
I20260810 11:42:32.560936 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30055
I20260810 11:42:32.562289 30142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:32.685110 30758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:32.683470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:37901 (local address 127.0.0.1:34064)
0810 11:42:32.684510 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:32.684690 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:37901: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383}
I20260810 11:42:32.685176 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30055
2026-08-10T11:42:32Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3152 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:32 dist-test-slave-drl0 krb5kdc[30814](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:32 dist-test-slave-drl0 krb5kdc[30814](info): set up 2 sockets
Aug 10 11:42:32 dist-test-slave-drl0 krb5kdc[30814](info): commencing operation
krb5kdc: starting...
W20260810 11:42:34.082243 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.345s user 0.007s sys 0.042s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:34 dist-test-slave-drl0 krb5kdc[30814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:42:34.446581 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42945
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42945
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:34.829087 30831 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:34.829422 30831 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:34.829530 30831 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:34.837577 30831 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:34.837699 30831 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:34.837749 30831 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:34.837800 30831 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:34.837837 30831 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:34.848883 30831 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42945
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42945
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:34.850279 30831 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:34.851596 30831 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:34.864540 30839 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:34.864745 30842 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:34.864601 30840 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:34.866771 30831 server_base.cc:1034] running on GCE node
I20260810 11:42:34.867697 30831 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:34.869470 30831 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:34.870793 30831 hybrid_clock.cc:648] HybridClock initialized: now 1786362154870714 us; error 91 us; skew 500 ppm
I20260810 11:42:34.871222 30831 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:34 dist-test-slave-drl0 krb5kdc[30814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:34.878687 30831 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:42:34.880474 30831 webserver.cc:460] Webserver started at http://127.3.240.254:38321/ using document root <none> and password file <none>
I20260810 11:42:34.881222 30831 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:34.881330 30831 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:34.881709 30831 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:34.884896 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "a1b40ca8991c476d9773a75af799da9b"
format_stamp: "Formatted at 2026-08-10 11:42:34 on dist-test-slave-drl0"
server_key: "d02e9b9e757f447d366d53e369a2a6e2"
server_key_iv: "1b815c2e31e5a5ec157181f2c8573b82"
server_key_version: "encryptionkey@0"
I20260810 11:42:34.885669 30831 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a1b40ca8991c476d9773a75af799da9b"
format_stamp: "Formatted at 2026-08-10 11:42:34 on dist-test-slave-drl0"
server_key: "d02e9b9e757f447d366d53e369a2a6e2"
server_key_iv: "1b815c2e31e5a5ec157181f2c8573b82"
server_key_version: "encryptionkey@0"
I20260810 11:42:34.891726 30831 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260810 11:42:34.895759 30851 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:34.897235 30831 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:42:34.897441 30831 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "a1b40ca8991c476d9773a75af799da9b"
format_stamp: "Formatted at 2026-08-10 11:42:34 on dist-test-slave-drl0"
server_key: "d02e9b9e757f447d366d53e369a2a6e2"
server_key_iv: "1b815c2e31e5a5ec157181f2c8573b82"
server_key_version: "encryptionkey@0"
I20260810 11:42:34.897662 30831 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:34.916952 30831 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:34.921482 30831 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:34.921804 30831 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:34.949021 30831 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42945
I20260810 11:42:34.949054 30922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42945 every 8 connection(s)
I20260810 11:42:34.950747 30831 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:34.954541 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30831
I20260810 11:42:34.954763 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:34.955370 30923 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:34.955519 4035 external_mini_cluster.cc:1428] Setting key fa04b1b45f556e571c4779c943888cc8
I20260810 11:42:34.972976 30923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: Bootstrap starting.
Aug 10 11:42:34 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:34.977213 30923 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:34.978622 30923 log.cc:824] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:34.982393 30923 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: No bootstrap required, opened a new log
I20260810 11:42:34.984191 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:34.957550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53092 (local address 127.3.240.254:42945)
0810 11:42:34.958425 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:34.958436 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:34.958496 (+ 60us) server_negotiation.cc:407] Connection header received
0810 11:42:34.961355 (+ 2859us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:34.961378 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:34.961762 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:34.962101 (+ 339us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:34.964334 (+ 2233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:34.965975 (+ 1641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:34.968144 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:34.968921 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:34.969080 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:34.974184 (+ 5104us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:34.974196 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:34.974214 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:34.974266 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:34.978543 (+ 4277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:34.979438 (+ 895us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:34.979447 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:34.979455 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:34.979645 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:34.980132 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:34.980136 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:34.980138 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:34.980881 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:34.981200 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:34.981654 (+ 454us) server_negotiation.cc:299] Negotiation successful
0810 11:42:34.982092 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":665,"thread_start_us":444,"threads_started":1}
I20260810 11:42:34.992380 30923 raft_consensus.cc:348] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } }
I20260810 11:42:34.992734 30923 raft_consensus.cc:374] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:34.992810 30923 raft_consensus.cc:729] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1b40ca8991c476d9773a75af799da9b, State: Initialized, Role: FOLLOWER
I20260810 11:42:34.993304 30923 consensus_queue.cc:260] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } }
I20260810 11:42:34.993505 30923 raft_consensus.cc:388] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:34.993597 30923 raft_consensus.cc:482] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:34.993777 30923 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:34.996939 30923 raft_consensus.cc:504] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } }
I20260810 11:42:34.997924 30923 leader_election.cc:302] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a1b40ca8991c476d9773a75af799da9b; no voters:
I20260810 11:42:34.998499 30923 leader_election.cc:288] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:34.998649 30929 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:34.999449 30929 raft_consensus.cc:686] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [term 1 LEADER]: Becoming Leader. State: Replica: a1b40ca8991c476d9773a75af799da9b, State: Running, Role: LEADER
I20260810 11:42:34.999962 30929 consensus_queue.cc:237] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } }
I20260810 11:42:35.000777 30923 sys_catalog.cc:564] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:35.003540 30930 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a1b40ca8991c476d9773a75af799da9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } } }
I20260810 11:42:35.003923 30930 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:35.004819 30939 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:35.004643 30932 sys_catalog.cc:455] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [sys.catalog]: SysCatalogTable state changed. Reason: New leader a1b40ca8991c476d9773a75af799da9b. Latest consensus state: current_term: 1 leader_uuid: "a1b40ca8991c476d9773a75af799da9b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a1b40ca8991c476d9773a75af799da9b" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42945 } } }
I20260810 11:42:35.005048 30932 sys_catalog.cc:458] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:35.013480 30939 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:35.025113 30939 catalog_manager.cc:1347] Generated new cluster ID: 8f736f7c0bd443988a8d440a93ed2f53
I20260810 11:42:35.025233 30939 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:35.037281 30939 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:35.038542 30939 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:35.051344 30939 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: Generated new TSK 0
I20260810 11:42:35.052366 30939 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:42:35.147123 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:35.525563 30964 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:35.525899 30964 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:35.526067 30964 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:35.533392 30964 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:35.533543 30964 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:35.533723 30964 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:35.543658 30964 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:35.544988 30964 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:35.546344 30964 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:35.561631 30972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:35.561582 30977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:35.561663 30971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:35.562844 30964 server_base.cc:1034] running on GCE node
I20260810 11:42:35.563925 30964 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:35.565361 30964 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:35.566676 30964 hybrid_clock.cc:648] HybridClock initialized: now 1786362155566616 us; error 47 us; skew 500 ppm
I20260810 11:42:35.567098 30964 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:35 dist-test-slave-drl0 krb5kdc[30814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:35.570652 30964 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:42:35.573256 30964 webserver.cc:460] Webserver started at http://127.3.240.193:46723/ using document root <none> and password file <none>
I20260810 11:42:35.574060 30964 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:35.574306 30964 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:35.574689 30964 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:35.577800 30964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "b08c24cff6f144d880fe001f684696c3"
format_stamp: "Formatted at 2026-08-10 11:42:35 on dist-test-slave-drl0"
server_key: "f4a42a5f049428cf30334d3cfdbcfbca"
server_key_iv: "f61bfa216dc1cce354442c2479f3bc02"
server_key_version: "encryptionkey@0"
I20260810 11:42:35.578601 30964 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "b08c24cff6f144d880fe001f684696c3"
format_stamp: "Formatted at 2026-08-10 11:42:35 on dist-test-slave-drl0"
server_key: "f4a42a5f049428cf30334d3cfdbcfbca"
server_key_iv: "f61bfa216dc1cce354442c2479f3bc02"
server_key_version: "encryptionkey@0"
I20260810 11:42:35.583959 30964 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260810 11:42:35.587718 30985 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:35.589160 30964 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:42:35.589535 30964 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "b08c24cff6f144d880fe001f684696c3"
format_stamp: "Formatted at 2026-08-10 11:42:35 on dist-test-slave-drl0"
server_key: "f4a42a5f049428cf30334d3cfdbcfbca"
server_key_iv: "f61bfa216dc1cce354442c2479f3bc02"
server_key_version: "encryptionkey@0"
I20260810 11:42:35.590368 30964 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:35.612670 30964 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:35.617331 30964 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:35.617589 30964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:35.618537 30964 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:35.619947 30964 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:35.620024 30964 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:35.620111 30964 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:35.620155 30964 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:35.660180 30964 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:44997
I20260810 11:42:35.660274 31152 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:44997 every 8 connection(s)
I20260810 11:42:35.661530 30964 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:35.664266 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 30964
I20260810 11:42:35.664479 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:35.665060 4035 external_mini_cluster.cc:1428] Setting key de8e00752ebe02e51a196716d796d1e0
Aug 10 11:42:35 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362155, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:35.685021 31161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:35.664297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:46757 (local address 127.3.240.254:42945)
0810 11:42:35.664843 (+ 546us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:35.664850 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:35.665855 (+ 1005us) server_negotiation.cc:407] Connection header received
0810 11:42:35.667800 (+ 1945us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:35.667807 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:35.667908 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:35.668033 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:35.669540 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:35.670681 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:35.672371 (+ 1690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:35.673062 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:35.673214 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:35.677646 (+ 4432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:35.677655 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:35.677662 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:35.677707 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:35.681417 (+ 3710us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:35.682474 (+ 1057us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:35.682479 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:35.682481 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:35.682604 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:35.683263 (+ 659us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:35.683267 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:35.683269 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:35.683549 (+ 280us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:35.683671 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:35.684590 (+ 919us) server_negotiation.cc:299] Negotiation successful
0810 11:42:35.684756 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":283,"threads_started":1}
I20260810 11:42:35.686520 31163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:35.664841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42945 (local address 127.3.240.193:46757)
0810 11:42:35.665693 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:35.665746 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:35.667541 (+ 1795us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:35.668259 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:35.668266 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:35.668784 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:35.669353 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:35.669370 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:35.670872 (+ 1502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:35.670876 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:35.672209 (+ 1333us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:35.672223 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:35.673307 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:35.673312 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:35.673545 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:35.674651 (+ 1106us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:35.674697 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:35.677417 (+ 2720us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:35.681635 (+ 4218us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:35.681647 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:35.681665 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:35.682317 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:35.682831 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:35.682839 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:35.682845 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:35.683113 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:35.683687 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:35.683698 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:35.684147 (+ 449us) client_negotiation.cc:769] Sending connection context
0810 11:42:35.684448 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 11:42:35.685024 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":588,"thread_start_us":330,"threads_started":1}
I20260810 11:42:35.688237 31154 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42945
I20260810 11:42:35.688669 31154 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:35.689527 31154 heartbeater.cc:511] Master 127.3.240.254:42945 requested a full tablet report, sending...
I20260810 11:42:35.691963 30877 ts_manager.cc:194] Registered new tserver with Master: b08c24cff6f144d880fe001f684696c3 (127.3.240.193:44997)
I20260810 11:42:35.693459 30877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:46757
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:42:35.747587 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:36.122354 31167 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:36.122685 31167 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:36.122862 31167 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:36.130738 31167 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:36.130904 31167 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:36.131103 31167 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:36.141335 31167 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:36.142673 31167 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:36.144063 31167 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:36.159035 31176 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:36.159081 31177 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:36.159063 31179 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:36.160295 31167 server_base.cc:1034] running on GCE node
I20260810 11:42:36.160926 31167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:36.161875 31167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:36.163061 31167 hybrid_clock.cc:648] HybridClock initialized: now 1786362156163001 us; error 45 us; skew 500 ppm
I20260810 11:42:36.163405 31167 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:36 dist-test-slave-drl0 krb5kdc[30814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:36.166541 31167 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:42:36.168427 31167 webserver.cc:460] Webserver started at http://127.3.240.194:36619/ using document root <none> and password file <none>
I20260810 11:42:36.169248 31167 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:36.169363 31167 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:36.169628 31167 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:36.172601 31167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "9920e981c64a45779958f403020ffb34"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "e1812d20886887836fb52f9716280839"
server_key_iv: "a3827791572dce0ddb6c7adae48e64c9"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.173331 31167 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "9920e981c64a45779958f403020ffb34"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "e1812d20886887836fb52f9716280839"
server_key_iv: "a3827791572dce0ddb6c7adae48e64c9"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.178473 31167 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260810 11:42:36.182258 31189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.183714 31167 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260810 11:42:36.183866 31167 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "9920e981c64a45779958f403020ffb34"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "e1812d20886887836fb52f9716280839"
server_key_iv: "a3827791572dce0ddb6c7adae48e64c9"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.184029 31167 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:36.200788 31167 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:36.205968 31167 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:36.206470 31167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:36.207700 31167 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:36.209159 31167 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:36.209241 31167 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.209333 31167 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:36.209396 31167 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.248117 31167 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:40599
I20260810 11:42:36.248167 31350 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:40599 every 8 connection(s)
I20260810 11:42:36.249554 31167 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:36.254305 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 31167
I20260810 11:42:36.254534 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:36.255246 4035 external_mini_cluster.cc:1428] Setting key cbab070aa242ada9459f05bd3c022213
Aug 10 11:42:36 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:36.279906 31359 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.255210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:45695 (local address 127.3.240.254:42945)
0810 11:42:36.256662 (+ 1452us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.256668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.257770 (+ 1102us) server_negotiation.cc:407] Connection header received
0810 11:42:36.260791 (+ 3021us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.260801 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.260891 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.261043 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:36.263528 (+ 2485us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.264620 (+ 1092us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.266636 (+ 2016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.267452 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.267614 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.272549 (+ 4935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:36.272557 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:36.272564 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:36.272710 (+ 146us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:36.275983 (+ 3273us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.277128 (+ 1145us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.277132 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.277134 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.277252 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.277854 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.277858 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.277860 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.278148 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:36.278273 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.279345 (+ 1072us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.279548 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1302,"thread_start_us":6301,"threads_started":1}
I20260810 11:42:36.281461 31357 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.254846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42945 (local address 127.3.240.194:45695)
0810 11:42:36.257573 (+ 2727us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:36.257645 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:36.260449 (+ 2804us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:36.261552 (+ 1103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:36.261565 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:36.262476 (+ 911us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:36.263348 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.263369 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.264875 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.264881 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.266414 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.266434 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.267756 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.267762 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.268006 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.268949 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:36.268981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:36.271930 (+ 2949us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:36.276172 (+ 4242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:36.276186 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:36.276207 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:36.276939 (+ 732us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:36.277370 (+ 431us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:36.277377 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:36.277384 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:36.277690 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:36.278291 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:36.278301 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:36.278821 (+ 520us) client_negotiation.cc:769] Sending connection context
0810 11:42:36.279216 (+ 395us) client_negotiation.cc:240] Negotiation successful
0810 11:42:36.279885 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2285,"thread_start_us":1923,"threads_started":1}
I20260810 11:42:36.283097 31352 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42945
I20260810 11:42:36.283425 31352 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:36.284173 31352 heartbeater.cc:511] Master 127.3.240.254:42945 requested a full tablet report, sending...
I20260810 11:42:36.285856 30877 ts_manager.cc:194] Registered new tserver with Master: 9920e981c64a45779958f403020ffb34 (127.3.240.194:40599)
I20260810 11:42:36.286759 30877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:45695
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:42:36.339373 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:42:36.696858 31154 heartbeater.cc:503] Master 127.3.240.254:42945 was elected leader, sending a full tablet report...
W20260810 11:42:36.711935 31362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:36.712486 31362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:36.712695 31362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:36.720204 31362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:36.720371 31362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:36.720548 31362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:36.731359 31362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42945
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:36.732980 31362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:36.734611 31362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:36.748974 31369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:36.749006 31371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:36.748936 31373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:36.749306 31362 server_base.cc:1034] running on GCE node
I20260810 11:42:36.750288 31362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:36.751385 31362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:36.752720 31362 hybrid_clock.cc:648] HybridClock initialized: now 1786362156752646 us; error 59 us; skew 500 ppm
I20260810 11:42:36.753068 31362 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:36 dist-test-slave-drl0 krb5kdc[30814](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:36.756812 31362 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:42:36.759153 31362 webserver.cc:460] Webserver started at http://127.3.240.195:39523/ using document root <none> and password file <none>
I20260810 11:42:36.760304 31362 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:36.760497 31362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:36.760931 31362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:36.764887 31362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "8634a2729a704f1e98b2598f19465371"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "6b9a8a1be519d6a2ee5a6b186e872c58"
server_key_iv: "b02307ab32272f2202548b1029166ed4"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.766412 31362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "8634a2729a704f1e98b2598f19465371"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "6b9a8a1be519d6a2ee5a6b186e872c58"
server_key_iv: "b02307ab32272f2202548b1029166ed4"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.772779 31362 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260810 11:42:36.777814 31384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.779711 31362 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:42:36.779924 31362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "8634a2729a704f1e98b2598f19465371"
format_stamp: "Formatted at 2026-08-10 11:42:36 on dist-test-slave-drl0"
server_key: "6b9a8a1be519d6a2ee5a6b186e872c58"
server_key_iv: "b02307ab32272f2202548b1029166ed4"
server_key_version: "encryptionkey@0"
I20260810 11:42:36.780179 31362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:36.798991 31362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:36.803859 31362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:36.804175 31362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:36.805353 31362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:36.806959 31362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:36.807063 31362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.807220 31362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:36.807296 31362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:36.848214 31362 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:42007
I20260810 11:42:36.848259 31545 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:42007 every 8 connection(s)
I20260810 11:42:36.849577 31362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:36.855890 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 31362
I20260810 11:42:36.856050 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:36.856860 4035 external_mini_cluster.cc:1428] Setting key 41b0a031cf33fc88c470413244ad0672
Aug 10 11:42:36 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362156, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:36.874643 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.852912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48949 (local address 127.3.240.254:42945)
0810 11:42:36.853465 (+ 553us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.853472 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.854568 (+ 1096us) server_negotiation.cc:407] Connection header received
0810 11:42:36.858126 (+ 3558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.858131 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.858234 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.858350 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:36.860468 (+ 2118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.861416 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.863140 (+ 1724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.863539 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.863637 (+ 98us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.867464 (+ 3827us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:36.867468 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:36.867472 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:36.867504 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:36.870151 (+ 2647us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.871332 (+ 1181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.871337 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.871341 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.871506 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.872465 (+ 959us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.872471 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.872473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.872989 (+ 516us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:36.873173 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.874152 (+ 979us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.874350 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":240,"threads_started":1}
I20260810 11:42:36.875540 31549 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.853468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42945 (local address 127.3.240.195:48949)
0810 11:42:36.854371 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:36.854421 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:36.857791 (+ 3370us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:36.858610 (+ 819us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:36.858617 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:36.859460 (+ 843us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:36.860301 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.860321 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.861615 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.861620 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.862893 (+ 1273us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.862906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.863766 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.863771 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.863988 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.864727 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:36.864756 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:36.867239 (+ 2483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:36.870299 (+ 3060us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:36.870306 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:36.870317 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:36.871144 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:36.871840 (+ 696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:36.871851 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:36.871857 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:36.872268 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:36.873202 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:36.873212 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:36.873666 (+ 454us) client_negotiation.cc:769] Sending connection context
0810 11:42:36.873933 (+ 267us) client_negotiation.cc:240] Negotiation successful
0810 11:42:36.874329 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":656,"thread_start_us":449,"threads_started":1}
I20260810 11:42:36.877216 31546 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42945
I20260810 11:42:36.877547 31546 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:36.878474 31546 heartbeater.cc:511] Master 127.3.240.254:42945 requested a full tablet report, sending...
I20260810 11:42:36.880295 30877 ts_manager.cc:194] Registered new tserver with Master: 8634a2729a704f1e98b2598f19465371 (127.3.240.195:42007)
I20260810 11:42:36.881151 30877 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:48949
I20260810 11:42:36.886479 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:36.908877 31548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.890912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53100 (local address 127.3.240.254:42945)
0810 11:42:36.891165 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.891173 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.891850 (+ 677us) server_negotiation.cc:407] Connection header received
0810 11:42:36.891907 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.891911 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.891973 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.892206 (+ 233us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:36.893890 (+ 1684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.894727 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.896271 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.896684 (+ 413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.896788 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.899912 (+ 3124us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:36.899917 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:36.899920 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:36.899951 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:36.905044 (+ 5093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.905904 (+ 860us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.905908 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.905910 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.906033 (+ 123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:36.906705 (+ 672us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:36.906711 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:36.906714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:36.907271 (+ 557us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:36.907608 (+ 337us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.908289 (+ 681us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.908491 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260810 11:42:36.913470 30877 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53100:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:36.921149 30877 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:36.978752 31565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.942057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:40599 (local address 127.0.0.1:55816)
0810 11:42:36.952606 (+ 10549us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:36.952663 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:36.952921 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:36.963107 (+ 10186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:36.963115 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:36.963161 (+ 46us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:36.963740 (+ 579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.963759 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.970058 (+ 6299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.970068 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.973471 (+ 3403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.973495 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.977659 (+ 4164us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.977671 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.978026 (+ 355us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.978095 (+ 69us) client_negotiation.cc:769] Sending connection context
0810 11:42:36.978216 (+ 121us) client_negotiation.cc:240] Negotiation successful
0810 11:42:36.978412 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10236,"thread_start_us":12987,"threads_started":1}
I20260810 11:42:36.980453 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.942510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55816 (local address 127.3.240.194:40599)
0810 11:42:36.962253 (+ 19743us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.962267 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.962311 (+ 44us) server_negotiation.cc:407] Connection header received
0810 11:42:36.962462 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.962474 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.962825 (+ 351us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.963095 (+ 270us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:36.965097 (+ 2002us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.969464 (+ 4367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.973808 (+ 4344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.976418 (+ 2610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.979409 (+ 2991us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.979733 (+ 324us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.979822 (+ 89us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.980065 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19491,"thread_start_us":19348,"threads_started":1}
I20260810 11:42:36.985600 31263 tablet_service.cc:1467] Processing CreateTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 (DEFAULT_TABLE table=test-table [id=be66960851b0489a97ad7b00f3d3acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:36.987486 31263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc74d8bcdbac4bee855f654d490e47b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:36.988523 31568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.943385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44997 (local address 127.0.0.1:33928)
0810 11:42:36.978014 (+ 34629us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:36.978029 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:36.978166 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:36.978881 (+ 715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:36.978888 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:36.978988 (+ 100us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:36.979583 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.979599 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.982557 (+ 2958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.982563 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.985307 (+ 2744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.985336 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.987586 (+ 2250us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.987592 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.987776 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.987822 (+ 46us) client_negotiation.cc:769] Sending connection context
0810 11:42:36.987902 (+ 80us) client_negotiation.cc:240] Negotiation successful
0810 11:42:36.988044 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34516,"mutex_wait_us":4,"thread_start_us":33440,"threads_started":1}
I20260810 11:42:36.989420 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.944374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33928 (local address 127.3.240.193:44997)
0810 11:42:36.970695 (+ 26321us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.970705 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.978155 (+ 7450us) server_negotiation.cc:407] Connection header received
0810 11:42:36.978315 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.978324 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.978631 (+ 307us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.978849 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:36.979856 (+ 1007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.982358 (+ 2502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.985518 (+ 3160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.987088 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.987214 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.987439 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.988895 (+ 1456us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.989079 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26105,"thread_start_us":26317,"threads_started":1}
I20260810 11:42:36.989615 31569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.942789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42007 (local address 127.0.0.1:48366)
0810 11:42:36.977250 (+ 34461us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:36.977279 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:36.977434 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:36.979202 (+ 1768us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:36.979206 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:36.979231 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:36.979899 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.979914 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.983099 (+ 3185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.983105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.986146 (+ 3041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:36.986166 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.988595 (+ 2429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.988604 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:36.988885 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.988922 (+ 37us) client_negotiation.cc:769] Sending connection context
0810 11:42:36.989017 (+ 95us) client_negotiation.cc:240] Negotiation successful
0810 11:42:36.989218 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34285,"mutex_wait_us":137,"thread_start_us":34951,"threads_started":1}
I20260810 11:42:36.990674 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:36.943531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48366 (local address 127.3.240.195:42007)
0810 11:42:36.971376 (+ 27845us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:36.971384 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:36.978307 (+ 6923us) server_negotiation.cc:407] Connection header received
0810 11:42:36.978409 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:36.978415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:36.978630 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:36.978849 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:36.980145 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.982867 (+ 2722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.986366 (+ 3499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:36.987948 (+ 1582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:36.988073 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:36.988381 (+ 308us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:36.990027 (+ 1646us) server_negotiation.cc:299] Negotiation successful
0810 11:42:36.990202 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27667,"thread_start_us":27843,"threads_started":1}
I20260810 11:42:36.995016 31052 tablet_service.cc:1467] Processing CreateTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 (DEFAULT_TABLE table=test-table [id=be66960851b0489a97ad7b00f3d3acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:36.996718 31052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc74d8bcdbac4bee855f654d490e47b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:37.007323 31572 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Bootstrap starting.
I20260810 11:42:37.001889 31448 tablet_service.cc:1467] Processing CreateTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 (DEFAULT_TABLE table=test-table [id=be66960851b0489a97ad7b00f3d3acf0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:37.008297 31448 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet cc74d8bcdbac4bee855f654d490e47b7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:37.010390 31572 tablet_bootstrap.cc:654] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:37.011787 31572 log.cc:824] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:37.016264 31572 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: No bootstrap required, opened a new log
I20260810 11:42:37.016580 31572 ts_tablet_manager.cc:1397] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260810 11:42:37.021613 31575 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Bootstrap starting.
I20260810 11:42:37.023449 31576 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Bootstrap starting.
I20260810 11:42:37.025583 31575 tablet_bootstrap.cc:654] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:37.025657 31572 raft_consensus.cc:348] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.026146 31572 raft_consensus.cc:374] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:37.026249 31572 raft_consensus.cc:729] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9920e981c64a45779958f403020ffb34, State: Initialized, Role: FOLLOWER
I20260810 11:42:37.026289 31576 tablet_bootstrap.cc:654] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:37.027022 31575 log.cc:824] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:37.027019 31572 consensus_queue.cc:260] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.027379 31576 log.cc:824] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:37.028491 31352 heartbeater.cc:503] Master 127.3.240.254:42945 was elected leader, sending a full tablet report...
I20260810 11:42:37.029191 31572 ts_tablet_manager.cc:1428] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Time spent starting tablet: real 0.012s user 0.013s sys 0.000s
I20260810 11:42:37.030828 31575 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: No bootstrap required, opened a new log
I20260810 11:42:37.030865 31576 tablet_bootstrap.cc:492] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: No bootstrap required, opened a new log
I20260810 11:42:37.031144 31575 ts_tablet_manager.cc:1397] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260810 11:42:37.031190 31576 ts_tablet_manager.cc:1397] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.003s
I20260810 11:42:37.039074 31575 raft_consensus.cc:348] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.039449 31575 raft_consensus.cc:374] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:37.039579 31575 raft_consensus.cc:729] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b08c24cff6f144d880fe001f684696c3, State: Initialized, Role: FOLLOWER
I20260810 11:42:37.039347 31576 raft_consensus.cc:348] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.039785 31576 raft_consensus.cc:374] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:37.039902 31576 raft_consensus.cc:729] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8634a2729a704f1e98b2598f19465371, State: Initialized, Role: FOLLOWER
I20260810 11:42:37.040207 31575 consensus_queue.cc:260] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.040380 31576 consensus_queue.cc:260] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.041509 31546 heartbeater.cc:503] Master 127.3.240.254:42945 was elected leader, sending a full tablet report...
I20260810 11:42:37.042115 31575 ts_tablet_manager.cc:1428] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260810 11:42:37.042130 31576 ts_tablet_manager.cc:1428] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20260810 11:42:37.101631 31547 tablet.cc:2369] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:37.149788 31577 raft_consensus.cc:482] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:37.150098 31577 raft_consensus.cc:504] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.151878 31577 leader_election.cc:288] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8634a2729a704f1e98b2598f19465371 (127.3.240.195:42007), b08c24cff6f144d880fe001f684696c3 (127.3.240.193:44997)
I20260810 11:42:37.160285 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.152439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:49421 (local address 127.3.240.193:44997)
0810 11:42:37.152680 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.152689 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.153540 (+ 851us) server_negotiation.cc:407] Connection header received
0810 11:42:37.153756 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.153762 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.153834 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.153906 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:37.154603 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.155503 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.157649 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.159130 (+ 1481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.159358 (+ 228us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.159458 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.159795 (+ 337us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.159952 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:42:37.160636 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.152406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:38177 (local address 127.3.240.195:42007)
0810 11:42:37.152648 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.152657 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.153591 (+ 934us) server_negotiation.cc:407] Connection header received
0810 11:42:37.153793 (+ 202us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.153797 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.153855 (+ 58us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.153964 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:37.154858 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.155749 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.158558 (+ 2809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.159618 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.159814 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.159902 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.160257 (+ 355us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.160383 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:42:37.161008 31591 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.152392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:44997 (local address 127.3.240.194:49421)
0810 11:42:37.153421 (+ 1029us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:37.153439 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:37.153594 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:37.153960 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:37.153964 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:37.153999 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:37.154450 (+ 451us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:37.154459 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.155641 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.155647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:37.157473 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:37.157486 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.159407 (+ 1921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.159413 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:37.159599 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.159628 (+ 29us) client_negotiation.cc:769] Sending connection context
0810 11:42:37.159748 (+ 120us) client_negotiation.cc:240] Negotiation successful
0810 11:42:37.159937 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"mutex_wait_us":23,"thread_start_us":737,"threads_started":1}
I20260810 11:42:37.161213 31587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.152415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:42007 (local address 127.3.240.194:38177)
0810 11:42:37.153455 (+ 1040us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:37.153471 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:37.153626 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:37.154012 (+ 386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:37.154016 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:37.154045 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:37.154637 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:37.154652 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.156059 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.156066 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:37.158306 (+ 2240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:37.158331 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.159887 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.159894 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:37.160072 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.160100 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 11:42:37.160223 (+ 123us) client_negotiation.cc:240] Negotiation successful
0810 11:42:37.160372 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":937,"mutex_wait_us":4,"thread_start_us":481,"threads_started":1}
I20260810 11:42:37.162400 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc74d8bcdbac4bee855f654d490e47b7" candidate_uuid: "9920e981c64a45779958f403020ffb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8634a2729a704f1e98b2598f19465371" is_pre_election: true
I20260810 11:42:37.162416 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc74d8bcdbac4bee855f654d490e47b7" candidate_uuid: "9920e981c64a45779958f403020ffb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08c24cff6f144d880fe001f684696c3" is_pre_election: true
I20260810 11:42:37.163059 31076 raft_consensus.cc:2454] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920e981c64a45779958f403020ffb34 in term 0.
I20260810 11:42:37.163059 31477 raft_consensus.cc:2454] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9920e981c64a45779958f403020ffb34 in term 0.
I20260810 11:42:37.163906 31192 leader_election.cc:302] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8634a2729a704f1e98b2598f19465371, 9920e981c64a45779958f403020ffb34; no voters:
W20260810 11:42:37.164466 31158 tablet.cc:2369] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:37.164466 31577 raft_consensus.cc:2781] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:37.164709 31577 raft_consensus.cc:482] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:37.164848 31577 raft_consensus.cc:3037] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:37.167724 31577 raft_consensus.cc:504] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.168331 31577 leader_election.cc:288] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [CANDIDATE]: Term 1 election: Requested vote from peers 8634a2729a704f1e98b2598f19465371 (127.3.240.195:42007), b08c24cff6f144d880fe001f684696c3 (127.3.240.193:44997)
I20260810 11:42:37.168943 31477 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc74d8bcdbac4bee855f654d490e47b7" candidate_uuid: "9920e981c64a45779958f403020ffb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8634a2729a704f1e98b2598f19465371"
I20260810 11:42:37.168934 31076 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "cc74d8bcdbac4bee855f654d490e47b7" candidate_uuid: "9920e981c64a45779958f403020ffb34" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b08c24cff6f144d880fe001f684696c3"
I20260810 11:42:37.169163 31477 raft_consensus.cc:3037] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:37.169164 31076 raft_consensus.cc:3037] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:37.172505 31076 raft_consensus.cc:2454] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920e981c64a45779958f403020ffb34 in term 1.
I20260810 11:42:37.172505 31477 raft_consensus.cc:2454] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9920e981c64a45779958f403020ffb34 in term 1.
I20260810 11:42:37.173065 31191 leader_election.cc:302] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9920e981c64a45779958f403020ffb34, b08c24cff6f144d880fe001f684696c3; no voters:
I20260810 11:42:37.173375 31577 raft_consensus.cc:2781] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:37.173926 31577 raft_consensus.cc:686] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 1 LEADER]: Becoming Leader. State: Replica: 9920e981c64a45779958f403020ffb34, State: Running, Role: LEADER
I20260810 11:42:37.174487 31577 consensus_queue.cc:237] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } }
I20260810 11:42:37.179275 30877 catalog_manager.cc:5495] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9920e981c64a45779958f403020ffb34 (127.3.240.194). New cstate: current_term: 1 leader_uuid: "9920e981c64a45779958f403020ffb34" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9920e981c64a45779958f403020ffb34" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 40599 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 } health_report { overall_health: UNKNOWN } } }
I20260810 11:42:37.207763 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.200756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55828 (local address 127.3.240.194:40599)
0810 11:42:37.201036 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.201043 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.201208 (+ 165us) server_negotiation.cc:407] Connection header received
0810 11:42:37.201524 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.201529 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.201637 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.201829 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:37.202471 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.203615 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.205206 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.205631 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.205772 (+ 141us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.206446 (+ 674us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:37.207014 (+ 568us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:37.207206 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.207383 (+ 177us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.207432 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260810 11:42:37.216037 31076 raft_consensus.cc:1261] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 1 FOLLOWER]: Refusing update from remote peer 9920e981c64a45779958f403020ffb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:37.216095 31477 raft_consensus.cc:1261] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 1 FOLLOWER]: Refusing update from remote peer 9920e981c64a45779958f403020ffb34: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:37.217041 31577 consensus_queue.cc:1035] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b08c24cff6f144d880fe001f684696c3" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 44997 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:37.217338 31592 consensus_queue.cc:1035] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8634a2729a704f1e98b2598f19465371" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 42007 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:37.230072 31596 mvcc.cc:205] Tried to move back new op lower bound from 7316939395947151360 to 7316939395791147008. Current Snapshot: MvccSnapshot[applied={T|T < 7316939395947151360}]
I20260810 11:42:37.246115 30876 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:37.246464 30876 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53100:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:37.251102 30876 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a1b40ca8991c476d9773a75af799da9b: Sending DeleteTablet for 3 replicas of tablet cc74d8bcdbac4bee855f654d490e47b7
I20260810 11:42:37.252295 31448 tablet_service.cc:1514] Processing DeleteTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:48366
I20260810 11:42:37.252347 31263 tablet_service.cc:1514] Processing DeleteTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:55816
I20260810 11:42:37.253612 31610 tablet_replica.cc:330] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: stopping tablet replica
I20260810 11:42:37.253893 31052 tablet_service.cc:1514] Processing DeleteTablet for tablet cc74d8bcdbac4bee855f654d490e47b7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:37 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:33928
I20260810 11:42:37.254467 31610 raft_consensus.cc:2229] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:37.255029 31610 raft_consensus.cc:2258] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:37.255952 31614 tablet_replica.cc:330] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: stopping tablet replica
I20260810 11:42:37.256378 31614 raft_consensus.cc:2229] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:37.256851 31614 raft_consensus.cc:2258] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:37.257575 31610 ts_tablet_manager.cc:1905] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:37.258725 31614 ts_tablet_manager.cc:1905] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:37.265180 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30964
I20260810 11:42:37.265690 31610 ts_tablet_manager.cc:1918] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:37.265877 31610 log.cc:1196] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/cc74d8bcdbac4bee855f654d490e47b7
I20260810 11:42:37.266630 31610 ts_tablet_manager.cc:1939] T cc74d8bcdbac4bee855f654d490e47b7 P 8634a2729a704f1e98b2598f19465371: Deleting consensus metadata
I20260810 11:42:37.268342 31614 ts_tablet_manager.cc:1918] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:37.268796 31614 log.cc:1196] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/cc74d8bcdbac4bee855f654d490e47b7
I20260810 11:42:37.269285 30854 catalog_manager.cc:4841] TS 8634a2729a704f1e98b2598f19465371 (127.3.240.195:42007): tablet cc74d8bcdbac4bee855f654d490e47b7 (table test-table [id=be66960851b0489a97ad7b00f3d3acf0]) successfully deleted
I20260810 11:42:37.270263 31614 ts_tablet_manager.cc:1939] T cc74d8bcdbac4bee855f654d490e47b7 P b08c24cff6f144d880fe001f684696c3: Deleting consensus metadata
I20260810 11:42:37.274861 30853 catalog_manager.cc:4841] TS b08c24cff6f144d880fe001f684696c3 (127.3.240.193:44997): tablet cc74d8bcdbac4bee855f654d490e47b7 (table test-table [id=be66960851b0489a97ad7b00f3d3acf0]) successfully deleted
I20260810 11:42:37.275810 31613 tablet_replica.cc:330] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: stopping tablet replica
I20260810 11:42:37.276239 31613 raft_consensus.cc:2229] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:37.276885 31613 raft_consensus.cc:2258] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:37.278982 31613 ts_tablet_manager.cc:1905] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 10 11:42:37 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:42:37.289063 31613 ts_tablet_manager.cc:1918] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:37.289247 31613 log.cc:1196] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/cc74d8bcdbac4bee855f654d490e47b7
I20260810 11:42:37.289943 31613 ts_tablet_manager.cc:1939] T cc74d8bcdbac4bee855f654d490e47b7 P 9920e981c64a45779958f403020ffb34: Deleting consensus metadata
I20260810 11:42:37.291636 30852 catalog_manager.cc:4841] TS 9920e981c64a45779958f403020ffb34 (127.3.240.194:40599): tablet cc74d8bcdbac4bee855f654d490e47b7 (table test-table [id=be66960851b0489a97ad7b00f3d3acf0]) successfully deleted
I20260810 11:42:37.295558 31571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.267394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33936 (local address 127.3.240.193:44997)
0810 11:42:37.267574 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.267580 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.269614 (+ 2034us) server_negotiation.cc:407] Connection header received
0810 11:42:37.270680 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.270686 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.270793 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.270872 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:37.274697 (+ 3825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.275877 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.278370 (+ 2493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.279380 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.279492 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.285763 (+ 6271us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:37.285783 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:37.285815 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:37.285885 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:37.291375 (+ 5490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.292641 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.292650 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.292653 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.292851 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.293598 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.293602 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.293605 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.294051 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:37.294301 (+ 250us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.294773 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.295079 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 11:42:37.296288 31145 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:37.399639 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30964
I20260810 11:42:37.433974 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 31167
Aug 10 11:42:37 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:42:37.455843 31567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.434781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.3.240.194:40599)
0810 11:42:37.435178 (+ 397us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.435188 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.435240 (+ 52us) server_negotiation.cc:407] Connection header received
0810 11:42:37.435351 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.435357 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.435488 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.435759 (+ 271us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:37.437128 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.438346 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.440162 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.440826 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.440996 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.446548 (+ 5552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:37.446569 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:37.446591 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:37.446637 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:37.451292 (+ 4655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.452086 (+ 794us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.452096 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.452100 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.452274 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.452862 (+ 588us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.452874 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.452876 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.453397 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:37.453794 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.454651 (+ 857us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.455084 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260810 11:42:37.456934 31346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:37.580183 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 31167
I20260810 11:42:37.611133 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 31362
Aug 10 11:42:37 dist-test-slave-drl0 krb5kdc[30814](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362154, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:42:37.628270 31570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:37.612036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48382 (local address 127.3.240.195:42007)
0810 11:42:37.612217 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:37.612222 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:37.612371 (+ 149us) server_negotiation.cc:407] Connection header received
0810 11:42:37.612641 (+ 270us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:37.612646 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:37.612726 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:37.612832 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:37.614266 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.615209 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.616510 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:37.616915 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:37.617035 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:37.621919 (+ 4884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:37.621936 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:37.621967 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:37.622006 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:37.625164 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.625893 (+ 729us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.625901 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.625903 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.626049 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:37.626520 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:37.626524 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:37.626526 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:37.627006 (+ 480us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:37.627231 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:37.627684 (+ 453us) server_negotiation.cc:299] Negotiation successful
0810 11:42:37.627894 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 11:42:37.628957 31537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:37.724360 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 31362
I20260810 11:42:37.749532 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 30831
I20260810 11:42:37.750730 30918 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:37.873865 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 30831
2026-08-10T11:42:37Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5193 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-10T11:42:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:37Z Disabled control of system clock
I20260810 11:42:37.983712 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39733
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39733
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:38.373171 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:38.373529 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:38.373679 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:38.381234 31628 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:38.381397 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:38.381482 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:38.381599 31628 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:38.381676 31628 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:38.391214 31628 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39733
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:38.392638 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:38.394502 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:38.406247 31634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:38.407061 31635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:38.408004 31637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:38.408785 31628 server_base.cc:1034] running on GCE node
I20260810 11:42:38.409670 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:38.411187 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:38.412472 31628 hybrid_clock.cc:648] HybridClock initialized: now 1786362158412432 us; error 26 us; skew 500 ppm
I20260810 11:42:38.412998 31628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:38.416646 31628 webserver.cc:460] Webserver started at http://127.3.240.254:42199/ using document root <none> and password file <none>
I20260810 11:42:38.417387 31628 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:38.417479 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:38.417850 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:38.420991 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "d1b61755bb9b46249d47d96d67cf87b3"
format_stamp: "Formatted at 2026-08-10 11:42:38 on dist-test-slave-drl0"
server_key: "904ffb5b070b1ca2085ec549a41dacfe"
server_key_iv: "571bd8f611c340a17ffee9e92f9c4d99"
server_key_version: "encryptionkey@0"
I20260810 11:42:38.421820 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "d1b61755bb9b46249d47d96d67cf87b3"
format_stamp: "Formatted at 2026-08-10 11:42:38 on dist-test-slave-drl0"
server_key: "904ffb5b070b1ca2085ec549a41dacfe"
server_key_iv: "571bd8f611c340a17ffee9e92f9c4d99"
server_key_version: "encryptionkey@0"
I20260810 11:42:38.429153 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260810 11:42:38.434063 31645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:38.435953 31628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:42:38.436269 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "d1b61755bb9b46249d47d96d67cf87b3"
format_stamp: "Formatted at 2026-08-10 11:42:38 on dist-test-slave-drl0"
server_key: "904ffb5b070b1ca2085ec549a41dacfe"
server_key_iv: "571bd8f611c340a17ffee9e92f9c4d99"
server_key_version: "encryptionkey@0"
I20260810 11:42:38.436549 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:38.468807 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:38.469800 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:38.470249 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:38.490163 31628 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39733
I20260810 11:42:38.490281 31720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39733 every 8 connection(s)
I20260810 11:42:38.492118 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:38.497249 31722 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:38.498314 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 31628
I20260810 11:42:38.498554 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:38.499167 4035 external_mini_cluster.cc:1428] Setting key ba65d1712d2136882274ef638e3786d4
I20260810 11:42:38.517006 31722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: Bootstrap starting.
I20260810 11:42:38.519485 31725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:38.501656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55802 (local address 127.3.240.254:39733)
0810 11:42:38.503352 (+ 1696us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:38.503372 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:38.503457 (+ 85us) server_negotiation.cc:407] Connection header received
0810 11:42:38.506638 (+ 3181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:38.506679 (+ 41us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:38.507623 (+ 944us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:38.508267 (+ 644us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:38.508938 (+ 671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:38.511229 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:38.513161 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:38.514225 (+ 1064us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:38.514848 (+ 623us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:38.516259 (+ 1411us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:38.516271 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:38.516293 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:38.517105 (+ 812us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:38.517232 (+ 127us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:38.517261 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:38.517410 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:38.517745 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 11:42:38.518114 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":953,"thread_start_us":634,"threads_started":1}
I20260810 11:42:38.521421 31722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:38.522617 31722 log.cc:824] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:38.525840 31722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: No bootstrap required, opened a new log
I20260810 11:42:38.533358 31722 raft_consensus.cc:348] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } }
I20260810 11:42:38.533703 31722 raft_consensus.cc:374] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:38.533767 31722 raft_consensus.cc:729] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d1b61755bb9b46249d47d96d67cf87b3, State: Initialized, Role: FOLLOWER
I20260810 11:42:38.534421 31722 consensus_queue.cc:260] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } }
I20260810 11:42:38.534649 31722 raft_consensus.cc:388] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:38.534744 31722 raft_consensus.cc:482] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:38.534881 31722 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:38.537251 31722 raft_consensus.cc:504] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } }
I20260810 11:42:38.537843 31722 leader_election.cc:302] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d1b61755bb9b46249d47d96d67cf87b3; no voters:
I20260810 11:42:38.538463 31722 leader_election.cc:288] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:38.538632 31728 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:38.539973 31728 raft_consensus.cc:686] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [term 1 LEADER]: Becoming Leader. State: Replica: d1b61755bb9b46249d47d96d67cf87b3, State: Running, Role: LEADER
I20260810 11:42:38.540570 31728 consensus_queue.cc:237] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } }
I20260810 11:42:38.540935 31722 sys_catalog.cc:564] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:38.543661 31729 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d1b61755bb9b46249d47d96d67cf87b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } } }
I20260810 11:42:38.544064 31729 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:38.544540 31730 sys_catalog.cc:455] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d1b61755bb9b46249d47d96d67cf87b3. Latest consensus state: current_term: 1 leader_uuid: "d1b61755bb9b46249d47d96d67cf87b3" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d1b61755bb9b46249d47d96d67cf87b3" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39733 } } }
I20260810 11:42:38.544838 31730 sys_catalog.cc:458] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:38.546540 31735 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:38.554658 31735 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:38.564875 31735 catalog_manager.cc:1347] Generated new cluster ID: b856526621cf4854a58591b7362ba462
I20260810 11:42:38.565001 31735 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:38.581518 31735 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:38.583046 31735 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:38.591887 31735 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: Generated new TSK 0
I20260810 11:42:38.593103 31735 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:38.598281 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--builtin_ntp_servers=127.3.240.212:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:38.959553 31751 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:38.959988 31751 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:38.960162 31751 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:38.967867 31751 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:38.968001 31751 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:38.968179 31751 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:38.977252 31751 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:38.978576 31751 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:38.980458 31751 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:38.996663 31758 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:38.996639 31761 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:38.997159 31759 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:38.997605 31751 server_base.cc:1034] running on GCE node
I20260810 11:42:38.998283 31751 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:39.000386 31751 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:39.001770 31751 hybrid_clock.cc:648] HybridClock initialized: now 1786362159001677 us; error 54 us; skew 500 ppm
I20260810 11:42:39.002297 31751 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:39.005044 31751 webserver.cc:460] Webserver started at http://127.3.240.193:37053/ using document root <none> and password file <none>
I20260810 11:42:39.005798 31751 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:39.005944 31751 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:39.006294 31751 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:39.009270 31751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "868ea4926831449bba50d858ef8e91af"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "3565cd7de1fe02c88cdd6b7067754025"
server_key_iv: "a840a868df1e1f9a4e2111239221346c"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.010092 31751 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "868ea4926831449bba50d858ef8e91af"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "3565cd7de1fe02c88cdd6b7067754025"
server_key_iv: "a840a868df1e1f9a4e2111239221346c"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.015645 31751 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260810 11:42:39.019476 31769 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.021054 31751 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260810 11:42:39.021196 31751 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "868ea4926831449bba50d858ef8e91af"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "3565cd7de1fe02c88cdd6b7067754025"
server_key_iv: "a840a868df1e1f9a4e2111239221346c"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.021466 31751 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:39.036945 31751 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:39.037868 31751 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:39.038180 31751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:39.039256 31751 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:39.040868 31751 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:39.040974 31751 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.041126 31751 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:39.041234 31751 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.080753 31751 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:38475
I20260810 11:42:39.080854 31930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:38475 every 8 connection(s)
I20260810 11:42:39.082119 31751 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:39.085863 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 31751
I20260810 11:42:39.086050 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:39.086529 4035 external_mini_cluster.cc:1428] Setting key 1f4fe757cbd428e2a6f7415a4d5f6a0f
I20260810 11:42:39.089475 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--builtin_ntp_servers=127.3.240.212:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:39.097261 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:39.085722 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:56889 (local address 127.3.240.254:39733)
0810 11:42:39.086300 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:39.086309 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:39.087855 (+ 1546us) server_negotiation.cc:407] Connection header received
0810 11:42:39.089650 (+ 1795us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:39.089658 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:39.089750 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:39.089874 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:39.090914 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.092213 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.093756 (+ 1543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.094440 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.094589 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:39.096276 (+ 1687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:39.096285 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:39.096292 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:39.096318 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:39.096369 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:39.096387 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:39.096486 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:39.096711 (+ 225us) server_negotiation.cc:299] Negotiation successful
0810 11:42:39.096820 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":278,"threads_started":1}
I20260810 11:42:39.098049 31938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:39.086372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39733 (local address 127.3.240.193:56889)
0810 11:42:39.087690 (+ 1318us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:39.087738 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:39.089379 (+ 1641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:39.090086 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:39.090094 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:39.090174 (+ 80us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:39.090732 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:39.090748 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.092377 (+ 1629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.092380 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:39.093603 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:39.093615 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.094763 (+ 1148us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.094774 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:39.095084 (+ 310us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:39.095868 (+ 784us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:39.095871 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:39.095913 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:39.095989 (+ 76us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:39.096497 (+ 508us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:39.096506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:39.096512 (+ 6us) client_negotiation.cc:769] Sending connection context
0810 11:42:39.096665 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 11:42:39.096959 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"thread_start_us":463,"threads_started":1}
I20260810 11:42:39.100263 31931 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39733
I20260810 11:42:39.100848 31931 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:39.102331 31931 heartbeater.cc:511] Master 127.3.240.254:39733 requested a full tablet report, sending...
I20260810 11:42:39.105077 31671 ts_manager.cc:194] Registered new tserver with Master: 868ea4926831449bba50d858ef8e91af (127.3.240.193:38475)
I20260810 11:42:39.107268 31671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:56889
W20260810 11:42:39.445770 31941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:39.446055 31941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:39.446141 31941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:39.453500 31941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:39.453604 31941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:39.453711 31941 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:39.463824 31941 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:39.465107 31941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:39.467226 31941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:39.480769 31956 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:39.480854 31951 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:39.481302 31950 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:39.482218 31941 server_base.cc:1034] running on GCE node
I20260810 11:42:39.483314 31941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:39.484681 31941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:39.485939 31941 hybrid_clock.cc:648] HybridClock initialized: now 1786362159485905 us; error 63 us; skew 500 ppm
I20260810 11:42:39.486284 31941 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:39.488708 31941 webserver.cc:460] Webserver started at http://127.3.240.194:40649/ using document root <none> and password file <none>
I20260810 11:42:39.489476 31941 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:39.489619 31941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:39.489975 31941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:39.493762 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "7d253924bcf847aea7022db2b2a25a11"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "1758ced943c8d6872e689dd478b633ef"
server_key_iv: "9cae888537c9021b1bb1380a972da3f5"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.494750 31941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "7d253924bcf847aea7022db2b2a25a11"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "1758ced943c8d6872e689dd478b633ef"
server_key_iv: "9cae888537c9021b1bb1380a972da3f5"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.503121 31941 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260810 11:42:39.508219 31962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.509680 31941 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:42:39.509888 31941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "7d253924bcf847aea7022db2b2a25a11"
format_stamp: "Formatted at 2026-08-10 11:42:39 on dist-test-slave-drl0"
server_key: "1758ced943c8d6872e689dd478b633ef"
server_key_iv: "9cae888537c9021b1bb1380a972da3f5"
server_key_version: "encryptionkey@0"
I20260810 11:42:39.510123 31941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:39.538196 31941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:39.539197 31941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:39.539533 31941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:39.540860 31941 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:39.542536 31941 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:39.542627 31941 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.542701 31941 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:39.542760 31941 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:39.582823 31941 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:36371
I20260810 11:42:39.582916 32121 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:36371 every 8 connection(s)
I20260810 11:42:39.584257 31941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:39.590476 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 31941
I20260810 11:42:39.590715 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:39.591549 4035 external_mini_cluster.cc:1428] Setting key 3d72e4f369e2fcad0442b7fe529c19c5
I20260810 11:42:39.595707 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--builtin_ntp_servers=127.3.240.212:43887
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:39.600744 31937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:39.587740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:59797 (local address 127.3.240.254:39733)
0810 11:42:39.588035 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:39.588042 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:39.589925 (+ 1883us) server_negotiation.cc:407] Connection header received
0810 11:42:39.593181 (+ 3256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:39.593188 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:39.593290 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:39.593409 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:39.595351 (+ 1942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.596208 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.597799 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.598210 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.598359 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:39.599869 (+ 1510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:39.599874 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:39.599877 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:39.599896 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:39.599925 (+ 29us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:39.599937 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:39.600037 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:39.600284 (+ 247us) server_negotiation.cc:299] Negotiation successful
0810 11:42:39.600394 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260810 11:42:39.601866 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:39.588261 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39733 (local address 127.3.240.194:59797)
0810 11:42:39.589666 (+ 1405us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:39.589735 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:39.592863 (+ 3128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:39.593696 (+ 833us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:39.593706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:39.594114 (+ 408us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:39.595155 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:39.595177 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.596376 (+ 1199us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.596383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:39.597642 (+ 1259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:39.597655 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:39.598472 (+ 817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:39.598480 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:39.598718 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:39.599528 (+ 810us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:39.599531 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:39.599573 (+ 42us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:39.599659 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:39.600039 (+ 380us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:39.600053 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:39.600064 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 11:42:39.600250 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 11:42:39.600670 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":363,"threads_started":1}
I20260810 11:42:39.603421 32122 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39733
I20260810 11:42:39.603781 32122 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:39.604754 32122 heartbeater.cc:511] Master 127.3.240.254:39733 requested a full tablet report, sending...
I20260810 11:42:39.606806 31671 ts_manager.cc:194] Registered new tserver with Master: 7d253924bcf847aea7022db2b2a25a11 (127.3.240.194:36371)
I20260810 11:42:39.608052 31671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:59797
W20260810 11:42:39.966456 32129 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:39.966842 32129 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:39.967039 32129 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:39.974660 32129 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:39.974824 32129 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:39.975036 32129 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:39.984596 32129 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:43887
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:39733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:39.986234 32129 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:39.988382 32129 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:40.003652 32138 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:40.003997 32136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:40.004038 32141 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:40.005326 32129 server_base.cc:1034] running on GCE node
I20260810 11:42:40.006096 32129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:40.007547 32129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:40.008945 32129 hybrid_clock.cc:648] HybridClock initialized: now 1786362160008858 us; error 59 us; skew 500 ppm
I20260810 11:42:40.009325 32129 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:40.015771 32129 webserver.cc:460] Webserver started at http://127.3.240.195:43041/ using document root <none> and password file <none>
I20260810 11:42:40.016600 32129 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:40.016693 32129 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:40.017037 32129 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:40.020215 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "5f933f6328e842bea1a2faef0c18123f"
format_stamp: "Formatted at 2026-08-10 11:42:40 on dist-test-slave-drl0"
server_key: "59141f068ae3117f7a27d1ee59620049"
server_key_iv: "bf715bd228d58c8f3d9e152a6d3b65c8"
server_key_version: "encryptionkey@0"
I20260810 11:42:40.021039 32129 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "5f933f6328e842bea1a2faef0c18123f"
format_stamp: "Formatted at 2026-08-10 11:42:40 on dist-test-slave-drl0"
server_key: "59141f068ae3117f7a27d1ee59620049"
server_key_iv: "bf715bd228d58c8f3d9e152a6d3b65c8"
server_key_version: "encryptionkey@0"
I20260810 11:42:40.026911 32129 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:42:40.032344 32147 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:40.034477 32129 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260810 11:42:40.034777 32129 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "5f933f6328e842bea1a2faef0c18123f"
format_stamp: "Formatted at 2026-08-10 11:42:40 on dist-test-slave-drl0"
server_key: "59141f068ae3117f7a27d1ee59620049"
server_key_iv: "bf715bd228d58c8f3d9e152a6d3b65c8"
server_key_version: "encryptionkey@0"
I20260810 11:42:40.035369 32129 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:40.050724 32129 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:40.051644 32129 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:40.051981 32129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:40.053097 32129 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:40.054750 32129 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:40.054880 32129 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:40.055042 32129 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:40.055135 32129 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:40.097354 32129 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:34299
I20260810 11:42:40.097447 32313 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:34299 every 8 connection(s)
I20260810 11:42:40.098706 32129 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:40.104275 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 32129
I20260810 11:42:40.104475 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:40.105204 4035 external_mini_cluster.cc:1428] Setting key 733e352ca0c93b55500dfbc473482a63
I20260810 11:42:40.110584 31931 heartbeater.cc:503] Master 127.3.240.254:39733 was elected leader, sending a full tablet report...
I20260810 11:42:40.115473 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.101723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:33005 (local address 127.3.240.254:39733)
0810 11:42:40.102323 (+ 600us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.102327 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.103648 (+ 1321us) server_negotiation.cc:407] Connection header received
0810 11:42:40.106358 (+ 2710us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.106364 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.106454 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.106599 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:40.108011 (+ 1412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.109111 (+ 1100us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.111279 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.111933 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.112070 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.114316 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:40.114321 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:40.114325 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:40.114343 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:40.114396 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:40.114408 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:40.114485 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.115016 (+ 531us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.115105 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":485,"thread_start_us":390,"threads_started":1}
I20260810 11:42:40.116977 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.102408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39733 (local address 127.3.240.195:33005)
0810 11:42:40.103475 (+ 1067us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.103513 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.105998 (+ 2485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.107050 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.107058 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.107142 (+ 84us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:40.107826 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.107843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.109294 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.109298 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.111112 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.111125 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.112420 (+ 1295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.112435 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.112831 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.113836 (+ 1005us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:40.113839 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:40.113895 (+ 56us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:40.114015 (+ 120us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:40.114567 (+ 552us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:40.114577 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:40.114587 (+ 10us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.114884 (+ 297us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.115505 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":324,"threads_started":1}
I20260810 11:42:40.118810 32314 heartbeater.cc:347] Connected to a master server at 127.3.240.254:39733
I20260810 11:42:40.119309 32314 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:40.120682 32314 heartbeater.cc:511] Master 127.3.240.254:39733 requested a full tablet report, sending...
I20260810 11:42:40.122812 31671 ts_manager.cc:194] Registered new tserver with Master: 5f933f6328e842bea1a2faef0c18123f (127.3.240.195:34299)
I20260810 11:42:40.123785 31671 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:33005
I20260810 11:42:40.135310 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:40.146778 32317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.140088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55814 (local address 127.3.240.254:39733)
0810 11:42:40.140453 (+ 365us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.140460 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.140800 (+ 340us) server_negotiation.cc:407] Connection header received
0810 11:42:40.141036 (+ 236us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.141043 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.141116 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.141230 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:40.141810 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.142532 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.144030 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.144436 (+ 406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.144579 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.145922 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:40.145927 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:40.145930 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:40.145951 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:40.145987 (+ 36us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:40.145999 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:40.146232 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.146427 (+ 195us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.146489 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260810 11:42:40.152001 31671 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:55814:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:40.159461 31671 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:40.199440 32329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.174355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36371 (local address 127.0.0.1:45238)
0810 11:42:40.175323 (+ 968us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.175361 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.175578 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.190398 (+ 14820us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.190403 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.190439 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.190946 (+ 507us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.190962 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.193009 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.193014 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.195893 (+ 2879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.195911 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.197969 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.197978 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.198233 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.198297 (+ 64us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.198603 (+ 306us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.198912 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":668,"mutex_wait_us":258,"thread_start_us":494,"threads_started":1}
I20260810 11:42:40.200029 32331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.174970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34299 (local address 127.0.0.1:55088)
0810 11:42:40.186487 (+ 11517us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.186513 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.186689 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.190580 (+ 3891us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.190585 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.190610 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.191288 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.191305 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.193728 (+ 2423us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.193734 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.196221 (+ 2487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.196238 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.198153 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.198159 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.199152 (+ 993us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.199190 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.199282 (+ 92us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.199447 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11314,"mutex_wait_us":780,"thread_start_us":11828,"threads_started":1}
I20260810 11:42:40.200328 32336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.176682 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38475 (local address 127.0.0.1:40096)
0810 11:42:40.187539 (+ 10857us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.187561 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.187692 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.189344 (+ 1652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.189350 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.189382 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.189762 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.189776 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.192772 (+ 2996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.192778 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.195781 (+ 3003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.195824 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.198665 (+ 2841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.198672 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.198959 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.199011 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.199122 (+ 111us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.199292 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10701,"mutex_wait_us":327,"thread_start_us":10510,"threads_started":1}
I20260810 11:42:40.200556 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.174561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45238 (local address 127.3.240.194:36371)
0810 11:42:40.189577 (+ 15016us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.189595 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.189625 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:42:40.189727 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.189735 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.189952 (+ 217us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.191169 (+ 1217us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.191439 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.192871 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.196158 (+ 3287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.197713 (+ 1555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.198946 (+ 1233us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.199291 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.199407 (+ 116us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.199809 (+ 402us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14833,"thread_start_us":15412,"threads_started":1}
I20260810 11:42:40.201009 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.175615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55088 (local address 127.3.240.195:34299)
0810 11:42:40.190103 (+ 14488us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.190110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.190144 (+ 34us) server_negotiation.cc:407] Connection header received
0810 11:42:40.190238 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.190243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.190381 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.190580 (+ 199us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.191511 (+ 931us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.193552 (+ 2041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.196473 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.197938 (+ 1465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.198242 (+ 304us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.198730 (+ 488us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.200412 (+ 1682us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.200622 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14325,"thread_start_us":14154,"threads_started":1}
I20260810 11:42:40.201560 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.176312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40096 (local address 127.3.240.193:38475)
0810 11:42:40.188476 (+ 12164us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.188503 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.188551 (+ 48us) server_negotiation.cc:407] Connection header received
0810 11:42:40.188690 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.188702 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.189046 (+ 344us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.189303 (+ 257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.189973 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.192444 (+ 2471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.196243 (+ 3799us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.198039 (+ 1796us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.198231 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.198575 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.199545 (+ 970us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.199996 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11793,"thread_start_us":11632,"threads_started":1}
I20260810 11:42:40.204921 31835 tablet_service.cc:1467] Processing CreateTablet for tablet de5ce36454c347619c222abefb10462c (DEFAULT_TABLE table=test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:40.205312 32215 tablet_service.cc:1467] Processing CreateTablet for tablet de5ce36454c347619c222abefb10462c (DEFAULT_TABLE table=test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:40.205670 32036 tablet_service.cc:1467] Processing CreateTablet for tablet de5ce36454c347619c222abefb10462c (DEFAULT_TABLE table=test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:40.206476 31835 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5ce36454c347619c222abefb10462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:40.206826 32215 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5ce36454c347619c222abefb10462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:40.207118 32036 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet de5ce36454c347619c222abefb10462c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:40.220896 32338 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Bootstrap starting.
I20260810 11:42:40.224841 32338 tablet_bootstrap.cc:654] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:40.226025 32338 log.cc:824] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:40.227850 32340 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Bootstrap starting.
I20260810 11:42:40.228395 32339 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Bootstrap starting.
I20260810 11:42:40.231014 32338 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: No bootstrap required, opened a new log
I20260810 11:42:40.231503 32338 ts_tablet_manager.cc:1397] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.009s
I20260810 11:42:40.231724 32340 tablet_bootstrap.cc:654] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:40.231748 32339 tablet_bootstrap.cc:654] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:40.232976 32340 log.cc:824] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:40.232976 32339 log.cc:824] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:40.235937 32340 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: No bootstrap required, opened a new log
I20260810 11:42:40.236218 32340 ts_tablet_manager.cc:1397] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260810 11:42:40.236423 32339 tablet_bootstrap.cc:492] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: No bootstrap required, opened a new log
I20260810 11:42:40.236747 32339 ts_tablet_manager.cc:1397] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260810 11:42:40.243031 32338 raft_consensus.cc:348] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.243536 32338 raft_consensus.cc:374] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:40.243637 32338 raft_consensus.cc:729] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 868ea4926831449bba50d858ef8e91af, State: Initialized, Role: FOLLOWER
I20260810 11:42:40.244372 32338 consensus_queue.cc:260] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.245430 32340 raft_consensus.cc:348] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.245886 32340 raft_consensus.cc:374] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:40.245982 32340 raft_consensus.cc:729] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f933f6328e842bea1a2faef0c18123f, State: Initialized, Role: FOLLOWER
I20260810 11:42:40.246475 32338 ts_tablet_manager.cc:1428] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Time spent starting tablet: real 0.015s user 0.011s sys 0.002s
I20260810 11:42:40.246660 32340 consensus_queue.cc:260] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.247812 32314 heartbeater.cc:503] Master 127.3.240.254:39733 was elected leader, sending a full tablet report...
I20260810 11:42:40.247779 32339 raft_consensus.cc:348] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.248245 32339 raft_consensus.cc:374] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:40.248368 32339 raft_consensus.cc:729] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d253924bcf847aea7022db2b2a25a11, State: Initialized, Role: FOLLOWER
I20260810 11:42:40.249469 32339 consensus_queue.cc:260] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.250294 32340 ts_tablet_manager.cc:1428] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260810 11:42:40.251971 32122 heartbeater.cc:503] Master 127.3.240.254:39733 was elected leader, sending a full tablet report...
I20260810 11:42:40.252560 32339 ts_tablet_manager.cc:1428] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
W20260810 11:42:40.335042 31935 tablet.cc:2369] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:42:40.336580 32123 tablet.cc:2369] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260810 11:42:40.350793 32315 tablet.cc:2369] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:40.352519 32346 raft_consensus.cc:482] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:40.352957 32346 raft_consensus.cc:504] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.355854 32346 leader_election.cc:288] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d253924bcf847aea7022db2b2a25a11 (127.3.240.194:36371), 868ea4926831449bba50d858ef8e91af (127.3.240.193:38475)
I20260810 11:42:40.359035 32345 raft_consensus.cc:482] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:40.359460 32345 raft_consensus.cc:504] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.362504 32318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.355895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36371 (local address 127.3.240.195:44685)
0810 11:42:40.356103 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.356125 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.356262 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.356857 (+ 595us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.356864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.356899 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.357105 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.357119 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.358455 (+ 1336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.358461 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.360415 (+ 1954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.360431 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.361521 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.361527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.361760 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.361804 (+ 44us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.361919 (+ 115us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.362079 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":43}
I20260810 11:42:40.363513 32330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.356121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:44685 (local address 127.3.240.194:36371)
0810 11:42:40.356310 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.356316 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.356344 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:42:40.356432 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.356436 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.356534 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.356640 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.357295 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.358311 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.360594 (+ 2283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.361353 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.361520 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.361595 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.362897 (+ 1302us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.363117 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260810 11:42:40.364604 32063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "5f933f6328e842bea1a2faef0c18123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d253924bcf847aea7022db2b2a25a11" is_pre_election: true
I20260810 11:42:40.365255 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.356610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:50787 (local address 127.3.240.193:38475)
0810 11:42:40.356808 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.356814 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.357137 (+ 323us) server_negotiation.cc:407] Connection header received
0810 11:42:40.357260 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.357265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.357349 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.357418 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.358150 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.359300 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.361971 (+ 2671us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.363723 (+ 1752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.363903 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.364008 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.364709 (+ 701us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.364915 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260810 11:42:40.365334 32063 raft_consensus.cc:2454] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f933f6328e842bea1a2faef0c18123f in term 0.
I20260810 11:42:40.366202 32348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.356315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:38475 (local address 127.3.240.195:50787)
0810 11:42:40.357024 (+ 709us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.357038 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.357135 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.357510 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.357515 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.357541 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.357949 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.357961 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.359489 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.359495 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.361678 (+ 2183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.361697 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.364194 (+ 2497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.364203 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.364473 (+ 270us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.364524 (+ 51us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.365465 (+ 941us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.365665 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":604,"mutex_wait_us":25,"thread_start_us":382,"threads_started":1}
I20260810 11:42:40.366273 32152 leader_election.cc:302] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f933f6328e842bea1a2faef0c18123f, 7d253924bcf847aea7022db2b2a25a11; no voters:
I20260810 11:42:40.366405 32345 leader_election.cc:288] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d253924bcf847aea7022db2b2a25a11 (127.3.240.194:36371), 5f933f6328e842bea1a2faef0c18123f (127.3.240.195:34299)
I20260810 11:42:40.367209 32346 raft_consensus.cc:2781] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:40.367419 32346 raft_consensus.cc:482] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:40.367571 32346 raft_consensus.cc:3037] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:40.369148 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "5f933f6328e842bea1a2faef0c18123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868ea4926831449bba50d858ef8e91af" is_pre_election: true
I20260810 11:42:40.369921 31866 raft_consensus.cc:2454] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5f933f6328e842bea1a2faef0c18123f in term 0.
I20260810 11:42:40.371511 32346 raft_consensus.cc:504] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.372323 32346 leader_election.cc:288] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [CANDIDATE]: Term 1 election: Requested vote from peers 7d253924bcf847aea7022db2b2a25a11 (127.3.240.194:36371), 868ea4926831449bba50d858ef8e91af (127.3.240.193:38475)
I20260810 11:42:40.372896 32063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "5f933f6328e842bea1a2faef0c18123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d253924bcf847aea7022db2b2a25a11"
I20260810 11:42:40.373133 32063 raft_consensus.cc:3037] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:40.373327 31866 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "5f933f6328e842bea1a2faef0c18123f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "868ea4926831449bba50d858ef8e91af"
I20260810 11:42:40.373684 31866 raft_consensus.cc:3037] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:40.376971 32063 raft_consensus.cc:2454] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f933f6328e842bea1a2faef0c18123f in term 1.
I20260810 11:42:40.377064 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.361952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:35685 (local address 127.3.240.194:36371)
0810 11:42:40.366972 (+ 5020us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.366980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.368214 (+ 1234us) server_negotiation.cc:407] Connection header received
0810 11:42:40.368509 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.368514 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.368590 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.368665 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.370058 (+ 1393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.371274 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.374031 (+ 2757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.375444 (+ 1413us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.375634 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.375720 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.376282 (+ 562us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.376416 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4786,"thread_start_us":5278,"threads_started":1}
I20260810 11:42:40.377617 32152 leader_election.cc:302] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f933f6328e842bea1a2faef0c18123f, 7d253924bcf847aea7022db2b2a25a11; no voters:
I20260810 11:42:40.377697 32350 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.361676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:36371 (local address 127.3.240.193:35685)
0810 11:42:40.368103 (+ 6427us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.368128 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.368331 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.368988 (+ 657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.368992 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.369020 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.369809 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.369828 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.371430 (+ 1602us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.371435 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.373773 (+ 2338us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.373789 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.375919 (+ 2130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.375925 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.376100 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.376133 (+ 33us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.377128 (+ 995us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.377282 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6193,"mutex_wait_us":68,"thread_start_us":5917,"threads_started":1}
I20260810 11:42:40.377908 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.367219 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:37573 (local address 127.3.240.195:34299)
0810 11:42:40.367409 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.367417 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.368477 (+ 1060us) server_negotiation.cc:407] Connection header received
0810 11:42:40.368716 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.368722 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.368811 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.368873 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:40.370059 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.371282 (+ 1223us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.375221 (+ 3939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.376576 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.376713 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.376776 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.377474 (+ 698us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.377598 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260810 11:42:40.378140 31866 raft_consensus.cc:2454] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f933f6328e842bea1a2faef0c18123f in term 1.
I20260810 11:42:40.378407 32353 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.366752 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:34299 (local address 127.3.240.193:37573)
0810 11:42:40.368383 (+ 1631us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:40.368407 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:40.368562 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:40.369032 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:40.369037 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:40.369062 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:40.369854 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.369865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.372128 (+ 2263us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.372133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.374978 (+ 2845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:40.374993 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.376957 (+ 1964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.376964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:40.377203 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.377237 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 11:42:40.377659 (+ 422us) client_negotiation.cc:240] Negotiation successful
0810 11:42:40.377897 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1407,"mutex_wait_us":39,"thread_start_us":1212,"threads_started":1}
I20260810 11:42:40.378422 32063 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "868ea4926831449bba50d858ef8e91af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d253924bcf847aea7022db2b2a25a11" is_pre_election: true
I20260810 11:42:40.378320 32346 raft_consensus.cc:2781] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:40.378742 32063 raft_consensus.cc:2379] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 868ea4926831449bba50d858ef8e91af in current term 1: Already voted for candidate 5f933f6328e842bea1a2faef0c18123f in this term.
I20260810 11:42:40.379508 32346 raft_consensus.cc:686] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 1 LEADER]: Becoming Leader. State: Replica: 5f933f6328e842bea1a2faef0c18123f, State: Running, Role: LEADER
I20260810 11:42:40.379460 32245 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "de5ce36454c347619c222abefb10462c" candidate_uuid: "868ea4926831449bba50d858ef8e91af" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5f933f6328e842bea1a2faef0c18123f" is_pre_election: true
I20260810 11:42:40.380107 32346 consensus_queue.cc:237] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } }
I20260810 11:42:40.380527 31770 leader_election.cc:302] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 868ea4926831449bba50d858ef8e91af; no voters: 5f933f6328e842bea1a2faef0c18123f, 7d253924bcf847aea7022db2b2a25a11
I20260810 11:42:40.380911 32345 raft_consensus.cc:2726] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260810 11:42:40.385026 31671 catalog_manager.cc:5495] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5f933f6328e842bea1a2faef0c18123f (127.3.240.195). New cstate: current_term: 1 leader_uuid: "5f933f6328e842bea1a2faef0c18123f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5f933f6328e842bea1a2faef0c18123f" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 34299 } health_report { overall_health: HEALTHY } } }
I20260810 11:42:40.449621 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.443138 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55094 (local address 127.3.240.195:34299)
0810 11:42:40.443382 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.443388 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.443407 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:42:40.443502 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.443505 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.443741 (+ 236us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.443890 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:40.444334 (+ 444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.445144 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.446869 (+ 1725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.447318 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.447443 (+ 125us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.448047 (+ 604us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:40.448606 (+ 559us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:40.448766 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.449254 (+ 488us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.449309 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260810 11:42:40.458797 32063 raft_consensus.cc:1261] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 1 FOLLOWER]: Refusing update from remote peer 5f933f6328e842bea1a2faef0c18123f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:40.458845 31866 raft_consensus.cc:1261] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 1 FOLLOWER]: Refusing update from remote peer 5f933f6328e842bea1a2faef0c18123f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:40.459884 32354 consensus_queue.cc:1035] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d253924bcf847aea7022db2b2a25a11" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 36371 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:40.460283 32346 consensus_queue.cc:1035] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [LEADER]: Connected to new peer: Peer: permanent_uuid: "868ea4926831449bba50d858ef8e91af" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 38475 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:40.492902 31669 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:40.493276 31669 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:55814:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:40.495651 32359 mvcc.cc:205] Tried to move back new op lower bound from 7316939409227161600 to 7316939408921890816. Current Snapshot: MvccSnapshot[applied={T|T < 7316939409227161600}]
I20260810 11:42:40.499644 31669 catalog_manager.cc:5782] T 00000000000000000000000000000000 P d1b61755bb9b46249d47d96d67cf87b3: Sending DeleteTablet for 3 replicas of tablet de5ce36454c347619c222abefb10462c
I20260810 11:42:40.501019 32215 tablet_service.cc:1514] Processing DeleteTablet for tablet de5ce36454c347619c222abefb10462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:40 UTC) from {username='slave'} at 127.0.0.1:55088
I20260810 11:42:40.501217 32036 tablet_service.cc:1514] Processing DeleteTablet for tablet de5ce36454c347619c222abefb10462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:40 UTC) from {username='slave'} at 127.0.0.1:45238
I20260810 11:42:40.501938 31835 tablet_service.cc:1514] Processing DeleteTablet for tablet de5ce36454c347619c222abefb10462c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:40 UTC) from {username='slave'} at 127.0.0.1:40096
I20260810 11:42:40.502140 32375 tablet_replica.cc:330] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: stopping tablet replica
I20260810 11:42:40.502589 32375 raft_consensus.cc:2229] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:40.503397 32375 raft_consensus.cc:2258] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:40.505774 32375 ts_tablet_manager.cc:1905] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:40.506598 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 31751
I20260810 11:42:40.511494 32376 tablet_replica.cc:330] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: stopping tablet replica
I20260810 11:42:40.511957 32376 raft_consensus.cc:2229] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:40.512501 32376 raft_consensus.cc:2258] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:40.517910 32376 ts_tablet_manager.cc:1905] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:40.520356 32377 tablet_replica.cc:330] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: stopping tablet replica
I20260810 11:42:40.521155 32377 raft_consensus.cc:2229] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:40.521714 32377 raft_consensus.cc:2258] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:40.524174 32377 ts_tablet_manager.cc:1905] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:40.525197 32376 ts_tablet_manager.cc:1918] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:40.525378 32376 log.cc:1196] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/de5ce36454c347619c222abefb10462c
I20260810 11:42:40.525945 32376 ts_tablet_manager.cc:1939] T de5ce36454c347619c222abefb10462c P 7d253924bcf847aea7022db2b2a25a11: Deleting consensus metadata
I20260810 11:42:40.533334 32377 ts_tablet_manager.cc:1918] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:40.533747 32377 log.cc:1196] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/de5ce36454c347619c222abefb10462c
I20260810 11:42:40.534564 32377 ts_tablet_manager.cc:1939] T de5ce36454c347619c222abefb10462c P 868ea4926831449bba50d858ef8e91af: Deleting consensus metadata
I20260810 11:42:40.535401 32375 ts_tablet_manager.cc:1918] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:40.535651 32375 log.cc:1196] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/de5ce36454c347619c222abefb10462c
I20260810 11:42:40.536269 32375 ts_tablet_manager.cc:1939] T de5ce36454c347619c222abefb10462c P 5f933f6328e842bea1a2faef0c18123f: Deleting consensus metadata
I20260810 11:42:40.537281 31649 catalog_manager.cc:4841] TS 868ea4926831449bba50d858ef8e91af (127.3.240.193:38475): tablet de5ce36454c347619c222abefb10462c (table test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]) successfully deleted
I20260810 11:42:40.538085 31647 catalog_manager.cc:4841] TS 5f933f6328e842bea1a2faef0c18123f (127.3.240.195:34299): tablet de5ce36454c347619c222abefb10462c (table test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]) successfully deleted
I20260810 11:42:40.539862 31649 catalog_manager.cc:4841] TS 7d253924bcf847aea7022db2b2a25a11 (127.3.240.194:36371): tablet de5ce36454c347619c222abefb10462c (table test-table [id=e94d7c0ebdff46f5ad64cfdaeea0b687]) successfully deleted
I20260810 11:42:40.540300 32334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.508469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40106 (local address 127.3.240.193:38475)
0810 11:42:40.508670 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.508676 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.527674 (+ 18998us) server_negotiation.cc:407] Connection header received
0810 11:42:40.527966 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.527972 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.528078 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.528148 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:40.528865 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.530140 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.532632 (+ 2492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.533395 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.533517 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.538651 (+ 5134us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:40.538660 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:40.538674 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:40.539437 (+ 763us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:40.539494 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:40.539515 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:40.539658 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.539822 (+ 164us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.539948 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 11:42:40.541651 31925 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:40.637985 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 31751
I20260810 11:42:40.675287 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 31941
I20260810 11:42:40.682663 32352 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.676245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45244 (local address 127.3.240.194:36371)
0810 11:42:40.676466 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.676474 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.676497 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:42:40.676567 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.676573 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.676712 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.676834 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:40.677288 (+ 454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.678194 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.679637 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.680036 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.680177 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.681302 (+ 1125us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:40.681317 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:40.681331 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:40.681922 (+ 591us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:40.681968 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:40.681988 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:40.682116 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.682254 (+ 138us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.682363 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 11:42:40.683408 32116 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:40.783990 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 31941
I20260810 11:42:40.812999 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 32129
I20260810 11:42:40.820806 32332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:40.813935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55104 (local address 127.3.240.195:34299)
0810 11:42:40.814135 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:40.814142 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:40.814165 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:42:40.814237 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:40.814243 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:40.814341 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:40.814460 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:40.815238 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.816056 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.817320 (+ 1264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:40.817809 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:40.817965 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:40.819230 (+ 1265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:40.819238 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:40.819256 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:40.819871 (+ 615us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:40.819922 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:40.819953 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:40.820048 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:40.820308 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 11:42:40.820396 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:42:40.821537 32309 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:40.963938 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 32129
I20260810 11:42:40.993908 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 31628
I20260810 11:42:40.995587 31716 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:41.112784 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 31628
2026-08-10T11:42:41Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3236 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:41 dist-test-slave-drl0 krb5kdc[32388](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:41 dist-test-slave-drl0 krb5kdc[32388](info): set up 2 sockets
Aug 10 11:42:41 dist-test-slave-drl0 krb5kdc[32388](info): commencing operation
krb5kdc: starting...
W20260810 11:42:42.492826 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.327s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:42 dist-test-slave-drl0 krb5kdc[32388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:42:42.864054 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42579
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:42579
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:43.263626 32413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:43.263921 32413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:43.264087 32413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:43.271716 32413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:43.271816 32413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:43.271914 32413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:43.272027 32413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:43.272081 32413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:43.285284 32413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:42579
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:42579
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:43.286677 32413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:43.288118 32413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:43.302255 32418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:43.302356 32424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:43.302381 32422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:43.303007 32413 server_base.cc:1034] running on GCE node
I20260810 11:42:43.303786 32413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:43.305191 32413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:43.306457 32413 hybrid_clock.cc:648] HybridClock initialized: now 1786362163306412 us; error 53 us; skew 500 ppm
I20260810 11:42:43.306835 32413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:43 dist-test-slave-drl0 krb5kdc[32388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362163, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:43.310372 32413 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:42:43.311882 32413 webserver.cc:460] Webserver started at http://127.3.240.254:41585/ using document root <none> and password file <none>
I20260810 11:42:43.312508 32413 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:43.312611 32413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:43.312855 32413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:43.315950 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "febd13dff2e64f6b918ffb54c4253056"
format_stamp: "Formatted at 2026-08-10 11:42:43 on dist-test-slave-drl0"
server_key: "9f6f99fd64a4440a047247c7d26a958b"
server_key_iv: "01736ea9a6ffb097d43bba445023ae16"
server_key_version: "encryptionkey@0"
I20260810 11:42:43.316725 32413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "febd13dff2e64f6b918ffb54c4253056"
format_stamp: "Formatted at 2026-08-10 11:42:43 on dist-test-slave-drl0"
server_key: "9f6f99fd64a4440a047247c7d26a958b"
server_key_iv: "01736ea9a6ffb097d43bba445023ae16"
server_key_version: "encryptionkey@0"
I20260810 11:42:43.323072 32413 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260810 11:42:43.326879 32433 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:43.328336 32413 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:42:43.328487 32413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "febd13dff2e64f6b918ffb54c4253056"
format_stamp: "Formatted at 2026-08-10 11:42:43 on dist-test-slave-drl0"
server_key: "9f6f99fd64a4440a047247c7d26a958b"
server_key_iv: "01736ea9a6ffb097d43bba445023ae16"
server_key_version: "encryptionkey@0"
I20260810 11:42:43.328657 32413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:43.351982 32413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:43.356374 32413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:43.356614 32413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:43.379884 32413 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:42579
I20260810 11:42:43.379948 32502 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:42579 every 8 connection(s)
I20260810 11:42:43.381302 32413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:43.385251 32505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:43.390401 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 32413
I20260810 11:42:43.390621 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:43.391211 4035 external_mini_cluster.cc:1428] Setting key b545b3d74e8e6e202e586dedf840bfa1
I20260810 11:42:43.398346 32505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: Bootstrap starting.
I20260810 11:42:43.401933 32505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:43.403213 32505 log.cc:824] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:43.406138 32505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: No bootstrap required, opened a new log
Aug 10 11:42:43 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:43.415257 32505 raft_consensus.cc:348] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } }
I20260810 11:42:43.416021 32505 raft_consensus.cc:374] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:43.416182 32505 raft_consensus.cc:729] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: febd13dff2e64f6b918ffb54c4253056, State: Initialized, Role: FOLLOWER
I20260810 11:42:43.417006 32505 consensus_queue.cc:260] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } }
I20260810 11:42:43.417263 32509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:43.393262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53908 (local address 127.3.240.254:42579)
0810 11:42:43.394118 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:43.394133 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:43.394182 (+ 49us) server_negotiation.cc:407] Connection header received
0810 11:42:43.396526 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:43.396550 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:43.396942 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:43.397367 (+ 425us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:43.399340 (+ 1973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:43.400762 (+ 1422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:43.402096 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:43.402849 (+ 753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:43.403110 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:43.407647 (+ 4537us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:43.407655 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:43.407669 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:43.407704 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:43.411356 (+ 3652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:43.412200 (+ 844us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:43.412207 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:43.412213 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:43.412393 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:43.412796 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:43.412801 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:43.412811 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:43.413348 (+ 537us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:43.413683 (+ 335us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:43.414160 (+ 477us) server_negotiation.cc:299] Negotiation successful
0810 11:42:43.415416 (+ 1256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":389,"threads_started":1}
I20260810 11:42:43.417353 32505 raft_consensus.cc:388] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:43.417685 32505 raft_consensus.cc:482] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:43.417958 32505 raft_consensus.cc:3037] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:43.421190 32505 raft_consensus.cc:504] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } }
I20260810 11:42:43.421725 32505 leader_election.cc:302] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: febd13dff2e64f6b918ffb54c4253056; no voters:
I20260810 11:42:43.422519 32505 leader_election.cc:288] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:43.422564 32512 raft_consensus.cc:2781] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:43.423588 32512 raft_consensus.cc:686] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [term 1 LEADER]: Becoming Leader. State: Replica: febd13dff2e64f6b918ffb54c4253056, State: Running, Role: LEADER
I20260810 11:42:43.424394 32512 consensus_queue.cc:237] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } }
I20260810 11:42:43.426383 32505 sys_catalog.cc:564] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:43.427878 32514 sys_catalog.cc:455] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "febd13dff2e64f6b918ffb54c4253056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } } }
I20260810 11:42:43.427897 32513 sys_catalog.cc:455] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [sys.catalog]: SysCatalogTable state changed. Reason: New leader febd13dff2e64f6b918ffb54c4253056. Latest consensus state: current_term: 1 leader_uuid: "febd13dff2e64f6b918ffb54c4253056" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "febd13dff2e64f6b918ffb54c4253056" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 42579 } } }
I20260810 11:42:43.428411 32513 sys_catalog.cc:458] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:43.429114 32514 sys_catalog.cc:458] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:43.429929 32518 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:43.434814 32518 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:43.444631 32518 catalog_manager.cc:1347] Generated new cluster ID: 282aec9f89ce4bd5afb5362aa14ee22f
I20260810 11:42:43.444748 32518 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:43.463946 32518 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:43.466249 32518 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:43.493546 32518 catalog_manager.cc:5868] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: Generated new TSK 0
I20260810 11:42:43.494707 32518 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.3.240.193@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.193@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
Entry for principal kudu/127.3.240.193 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab.
I20260810 11:42:43.586261 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:43.964589 32541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:43.964872 32541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:43.964974 32541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:43.973682 32541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:43.973802 32541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:43.973928 32541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:43.984011 32541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.193.keytab
--principal=kudu/127.3.240.193
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:43.985263 32541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:43.987591 32541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:44.002416 32549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:44.002370 32552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:44.002322 32550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:44.008910 32541 server_base.cc:1034] running on GCE node
I20260810 11:42:44.009929 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:44.011415 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:44.012777 32541 hybrid_clock.cc:648] HybridClock initialized: now 1786362164012622 us; error 128 us; skew 500 ppm
I20260810 11:42:44.013283 32541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:44 dist-test-slave-drl0 krb5kdc[32388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:44.016811 32541 init.cc:377] Logged in from keytab as kudu/127.3.240.193@KRBTEST.COM (short username kudu)
I20260810 11:42:44.018469 32541 webserver.cc:460] Webserver started at http://127.3.240.193:33581/ using document root <none> and password file <none>
I20260810 11:42:44.019289 32541 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:44.019411 32541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:44.019788 32541 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:44.023074 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "dfdd78ca0402451799c80e188b9c482b"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "5530a9abd25efe7885cc27c3b28a4cf4"
server_key_iv: "a296306d2b4e5709f24c5b8b80b1a77b"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.023926 32541 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "dfdd78ca0402451799c80e188b9c482b"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "5530a9abd25efe7885cc27c3b28a4cf4"
server_key_iv: "a296306d2b4e5709f24c5b8b80b1a77b"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.029393 32541 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260810 11:42:44.033324 32566 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.035054 32541 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260810 11:42:44.035233 32541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "dfdd78ca0402451799c80e188b9c482b"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "5530a9abd25efe7885cc27c3b28a4cf4"
server_key_iv: "a296306d2b4e5709f24c5b8b80b1a77b"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.035429 32541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:44.064705 32541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:44.069979 32541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:44.070323 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:44.071527 32541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:44.073066 32541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:44.073143 32541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.073267 32541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:44.073352 32541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.111939 32541 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:36299
I20260810 11:42:44.112141 32718 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:36299 every 8 connection(s)
I20260810 11:42:44.113495 32541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:44.114017 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 32541
I20260810 11:42:44.114210 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:44.114897 4035 external_mini_cluster.cc:1428] Setting key 7f1a8381f874d452afe60de998a066de
Aug 10 11:42:44 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.193@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:44.140460 32725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:44.116667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:59945 (local address 127.3.240.254:42579)
0810 11:42:44.117309 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:44.117315 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:44.118407 (+ 1092us) server_negotiation.cc:407] Connection header received
0810 11:42:44.121372 (+ 2965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:44.121376 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:44.121442 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:44.121538 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:44.123627 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.124360 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.127021 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.127519 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.127632 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:44.132464 (+ 4832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:44.132469 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:44.132472 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:44.132506 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:44.136803 (+ 4297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:44.137610 (+ 807us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:44.137618 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:44.137625 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:44.137775 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:44.138264 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:44.138271 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:44.138277 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:44.138727 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:44.138897 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:44.139836 (+ 939us) server_negotiation.cc:299] Negotiation successful
0810 11:42:44.140086 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520,"thread_start_us":244,"threads_started":1}
I20260810 11:42:44.141501 32726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:44.117336 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42579 (local address 127.3.240.193:59945)
0810 11:42:44.118207 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:44.118264 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:44.121041 (+ 2777us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:44.121804 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:44.121817 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:44.122581 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:44.123420 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:44.123448 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.124540 (+ 1092us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.124548 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:44.126741 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:44.126760 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.127756 (+ 996us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.127766 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:44.128099 (+ 333us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:44.128842 (+ 743us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:44.128876 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:44.132206 (+ 3330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:44.136999 (+ 4793us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:44.137006 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:44.137018 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:44.137457 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:44.137935 (+ 478us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:44.137939 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:44.137941 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:44.138131 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:44.138888 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:44.138895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:44.139329 (+ 434us) client_negotiation.cc:769] Sending connection context
0810 11:42:44.139659 (+ 330us) client_negotiation.cc:240] Negotiation successful
0810 11:42:44.140184 (+ 525us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":569,"thread_start_us":314,"threads_started":1}
I20260810 11:42:44.143875 32721 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42579
I20260810 11:42:44.144223 32721 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:44.144992 32721 heartbeater.cc:511] Master 127.3.240.254:42579 requested a full tablet report, sending...
I20260810 11:42:44.147430 32460 ts_manager.cc:194] Registered new tserver with Master: dfdd78ca0402451799c80e188b9c482b (127.3.240.193:36299)
I20260810 11:42:44.148903 32460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.193@KRBTEST.COM'} at 127.3.240.193:59945
WARNING: no policy specified for kudu/127.3.240.194@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.194@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
Entry for principal kudu/127.3.240.194 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab.
I20260810 11:42:44.198485 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:44.556563 32730 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:44.556876 32730 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:44.557044 32730 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:44.564404 32730 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:44.564545 32730 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:44.565008 32730 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:44.577356 32730 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.194.keytab
--principal=kudu/127.3.240.194
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:44.578848 32730 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:44.580255 32730 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:44.594048 32739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:44.594214 32741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:44.594019 32743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:44.595451 32730 server_base.cc:1034] running on GCE node
I20260810 11:42:44.596086 32730 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:44.597038 32730 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:44.598321 32730 hybrid_clock.cc:648] HybridClock initialized: now 1786362164598227 us; error 80 us; skew 500 ppm
I20260810 11:42:44.598613 32730 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:44 dist-test-slave-drl0 krb5kdc[32388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:44.602044 32730 init.cc:377] Logged in from keytab as kudu/127.3.240.194@KRBTEST.COM (short username kudu)
I20260810 11:42:44.603819 32730 webserver.cc:460] Webserver started at http://127.3.240.194:45995/ using document root <none> and password file <none>
I20260810 11:42:44.604485 32730 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:44.604616 32730 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:44.604949 32730 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:44.608963 32730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "68ff5958f9aa4e14979470873c47a904"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "adbedcbb2c3baee53101b58459dae199"
server_key_iv: "d50c0f010723415f21905d577f2a65fc"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.609892 32730 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "68ff5958f9aa4e14979470873c47a904"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "adbedcbb2c3baee53101b58459dae199"
server_key_iv: "d50c0f010723415f21905d577f2a65fc"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.616551 32730 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 11:42:44.620292 32752 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.621730 32730 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260810 11:42:44.621898 32730 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "68ff5958f9aa4e14979470873c47a904"
format_stamp: "Formatted at 2026-08-10 11:42:44 on dist-test-slave-drl0"
server_key: "adbedcbb2c3baee53101b58459dae199"
server_key_iv: "d50c0f010723415f21905d577f2a65fc"
server_key_version: "encryptionkey@0"
I20260810 11:42:44.622095 32730 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:44.646624 32730 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:44.652197 32730 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:44.652689 32730 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:44.654157 32730 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:44.656198 32730 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:44.656298 32730 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.656462 32730 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:44.656526 32730 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:44.693861 32730 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:37623
I20260810 11:42:44.693899 453 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:37623 every 8 connection(s)
I20260810 11:42:44.695314 32730 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:44.703228 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 32730
I20260810 11:42:44.703513 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:44.704106 4035 external_mini_cluster.cc:1428] Setting key 8794f691061184cf1b2b9fae73f0cbb3
Aug 10 11:42:44 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362164, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.194@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:44.718797 457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:44.698286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:33293 (local address 127.3.240.254:42579)
0810 11:42:44.698768 (+ 482us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:44.698772 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:44.699737 (+ 965us) server_negotiation.cc:407] Connection header received
0810 11:42:44.701895 (+ 2158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:44.701911 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:44.702002 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:44.702128 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:44.703787 (+ 1659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.704618 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.706568 (+ 1950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.707113 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.707252 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:44.712506 (+ 5254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:44.712511 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:44.712514 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:44.712561 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:44.715682 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:44.716478 (+ 796us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:44.716482 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:44.716485 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:44.716601 (+ 116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:44.717033 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:44.717040 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:44.717047 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:44.717321 (+ 274us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:44.717488 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:44.718179 (+ 691us) server_negotiation.cc:299] Negotiation successful
0810 11:42:44.718407 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":373,"thread_start_us":249,"threads_started":1}
I20260810 11:42:44.719940 461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:44.698820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42579 (local address 127.3.240.194:33293)
0810 11:42:44.699572 (+ 752us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:44.699617 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:44.701608 (+ 1991us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:44.702359 (+ 751us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:44.702366 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:44.702890 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:44.703603 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:44.703643 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.704782 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.704787 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:44.706410 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:44.706424 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:44.707350 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:44.707358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:44.707576 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:44.708812 (+ 1236us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:44.708848 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:44.712239 (+ 3391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:44.715830 (+ 3591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:44.715840 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:44.715859 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:44.716333 (+ 474us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:44.716713 (+ 380us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:44.716717 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:44.716721 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:44.716911 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:44.717458 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:44.717466 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:44.717802 (+ 336us) client_negotiation.cc:769] Sending connection context
0810 11:42:44.718084 (+ 282us) client_negotiation.cc:240] Negotiation successful
0810 11:42:44.718615 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":289,"threads_started":1}
I20260810 11:42:44.721848 454 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42579
I20260810 11:42:44.722173 454 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:44.722877 454 heartbeater.cc:511] Master 127.3.240.254:42579 requested a full tablet report, sending...
I20260810 11:42:44.724442 32460 ts_manager.cc:194] Registered new tserver with Master: 68ff5958f9aa4e14979470873c47a904 (127.3.240.194:37623)
I20260810 11:42:44.725256 32460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.194@KRBTEST.COM'} at 127.3.240.194:33293
WARNING: no policy specified for kudu/127.3.240.195@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.195@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
Entry for principal kudu/127.3.240.195 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab.
I20260810 11:42:44.794979 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:42:45.152238 32721 heartbeater.cc:503] Master 127.3.240.254:42579 was elected leader, sending a full tablet report...
W20260810 11:42:45.204945 464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:45.205255 464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:45.205425 464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:45.213337 464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:45.213472 464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:45.213712 464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:45.225848 464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.195.keytab
--principal=kudu/127.3.240.195
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:42579
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:45.227771 464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:45.229203 464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:45.245064 474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:45.245070 473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:45.245137 476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:45.246500 464 server_base.cc:1034] running on GCE node
I20260810 11:42:45.247227 464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:45.248262 464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:45.249497 464 hybrid_clock.cc:648] HybridClock initialized: now 1786362165249451 us; error 31 us; skew 500 ppm
I20260810 11:42:45.249841 464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:45 dist-test-slave-drl0 krb5kdc[32388](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362165, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:45.258558 464 init.cc:377] Logged in from keytab as kudu/127.3.240.195@KRBTEST.COM (short username kudu)
I20260810 11:42:45.260388 464 webserver.cc:460] Webserver started at http://127.3.240.195:42063/ using document root <none> and password file <none>
I20260810 11:42:45.261456 464 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:45.261668 464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:45.262048 464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:45.265709 464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "a38f775ee66e45e8a1d0781b794ad480"
format_stamp: "Formatted at 2026-08-10 11:42:45 on dist-test-slave-drl0"
server_key: "1d5a183dd6aec69a64961d630e8fd537"
server_key_iv: "e143be2ea8cc8fb6c2e0f28faf95513c"
server_key_version: "encryptionkey@0"
I20260810 11:42:45.266578 464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "a38f775ee66e45e8a1d0781b794ad480"
format_stamp: "Formatted at 2026-08-10 11:42:45 on dist-test-slave-drl0"
server_key: "1d5a183dd6aec69a64961d630e8fd537"
server_key_iv: "e143be2ea8cc8fb6c2e0f28faf95513c"
server_key_version: "encryptionkey@0"
I20260810 11:42:45.271991 464 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 11:42:45.276234 486 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:45.278044 464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260810 11:42:45.278261 464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "a38f775ee66e45e8a1d0781b794ad480"
format_stamp: "Formatted at 2026-08-10 11:42:45 on dist-test-slave-drl0"
server_key: "1d5a183dd6aec69a64961d630e8fd537"
server_key_iv: "e143be2ea8cc8fb6c2e0f28faf95513c"
server_key_version: "encryptionkey@0"
I20260810 11:42:45.278502 464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:45.304975 464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:45.311214 464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:45.311615 464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:45.313202 464 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:45.315160 464 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:45.315315 464 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:45.315475 464 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:45.315534 464 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:45.359452 464 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:33725
I20260810 11:42:45.359593 646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:33725 every 8 connection(s)
I20260810 11:42:45.361385 464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:45.370196 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 464
I20260810 11:42:45.370402 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:45.371182 4035 external_mini_cluster.cc:1428] Setting key 37703217fc84ecb04ebc374924a5ff1d
Aug 10 11:42:45 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362165, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.195@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:42:45.387976 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.365400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:42579 (local address 127.3.240.195:43169)
0810 11:42:45.366395 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.366444 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.368264 (+ 1820us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.369159 (+ 895us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.369167 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.369802 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:45.370673 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.370702 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.372063 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.372075 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.373600 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.373613 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.374460 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.374469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.374693 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.375483 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:42:45.375519 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:45.378036 (+ 2517us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:45.382275 (+ 4239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:45.382287 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:45.382313 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:45.383133 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:45.383603 (+ 470us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:42:45.383609 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:42:45.383612 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:42:45.383834 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:42:45.384540 (+ 706us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:45.384549 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:45.385224 (+ 675us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.385717 (+ 493us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.386566 (+ 849us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":749,"thread_start_us":433,"threads_started":1}
I20260810 11:42:45.389669 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.364782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:43169 (local address 127.3.240.254:42579)
0810 11:42:45.365261 (+ 479us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.365267 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.366565 (+ 1298us) server_negotiation.cc:407] Connection header received
0810 11:42:45.368684 (+ 2119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.368689 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.368773 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.368877 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:45.370905 (+ 2028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.371891 (+ 986us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.373770 (+ 1879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.374212 (+ 442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.374353 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.378252 (+ 3899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:45.378257 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:45.378261 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:45.378295 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:45.382041 (+ 3746us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.383284 (+ 1243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.383295 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.383301 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.383470 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.383975 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.383985 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.383987 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.384362 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:45.384537 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.389005 (+ 4468us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.389295 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354,"thread_start_us":219,"threads_started":1}
I20260810 11:42:45.390836 648 heartbeater.cc:347] Connected to a master server at 127.3.240.254:42579
I20260810 11:42:45.391220 648 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:45.392021 648 heartbeater.cc:511] Master 127.3.240.254:42579 requested a full tablet report, sending...
I20260810 11:42:45.393561 32460 ts_manager.cc:194] Registered new tserver with Master: a38f775ee66e45e8a1d0781b794ad480 (127.3.240.195:33725)
I20260810 11:42:45.394426 32460 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.3.240.195@KRBTEST.COM'} at 127.3.240.195:43169
I20260810 11:42:45.399302 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:45.419574 652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.404268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53918 (local address 127.3.240.254:42579)
0810 11:42:45.404627 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.404634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.404965 (+ 331us) server_negotiation.cc:407] Connection header received
0810 11:42:45.405132 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.405137 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.405223 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.405327 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:45.406921 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.407744 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.409132 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.409838 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.409958 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.412758 (+ 2800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:45.412763 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:45.412767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:45.412799 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:45.416164 (+ 3365us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.417245 (+ 1081us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.417249 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.417252 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.417400 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.418086 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.418090 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.418092 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.418441 (+ 349us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:45.418634 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.419153 (+ 519us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.419338 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:42:45.424310 32460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:45.432085 32460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:45.456705 667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.446075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:37623 (local address 127.0.0.1:41514)
0810 11:42:45.446874 (+ 799us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.446974 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.447183 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.448550 (+ 1367us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.448560 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.448595 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:45.449517 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.449532 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.451449 (+ 1917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.451455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.453609 (+ 2154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.453623 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.455477 (+ 1854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.455484 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.455830 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.455910 (+ 80us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.456054 (+ 144us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.456252 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"mutex_wait_us":62,"thread_start_us":391,"threads_started":1}
I20260810 11:42:45.457105 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.446112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41514 (local address 127.3.240.194:37623)
0810 11:42:45.447584 (+ 1472us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.447593 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.447623 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:42:45.447734 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.447743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.448011 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.448233 (+ 222us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:45.449739 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.451278 (+ 1539us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.453782 (+ 2504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.454857 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.455089 (+ 232us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.455317 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.456117 (+ 800us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.456493 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1318,"thread_start_us":1954,"threads_started":1}
I20260810 11:42:45.458204 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.446586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46896 (local address 127.3.240.193:36299)
0810 11:42:45.449209 (+ 2623us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.449217 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.449245 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:42:45.449335 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.449342 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.449511 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.449694 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:45.450356 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.452032 (+ 1676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.455276 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.457076 (+ 1800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.457243 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.457488 (+ 245us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.457720 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.457884 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2418,"thread_start_us":2192,"threads_started":1}
I20260810 11:42:45.458245 668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.446106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36299 (local address 127.0.0.1:46896)
0810 11:42:45.447797 (+ 1691us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.447813 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.447964 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.449689 (+ 1725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.449693 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.449717 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:45.450147 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.450157 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.452219 (+ 2062us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.452226 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.455074 (+ 2848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.455091 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.457324 (+ 2233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.457331 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.457527 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.457561 (+ 34us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.457664 (+ 103us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.457815 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1581,"mutex_wait_us":1,"thread_start_us":1365,"threads_started":1}
I20260810 11:42:45.459198 673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.448382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54154 (local address 127.3.240.195:33725)
0810 11:42:45.449540 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.449548 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.449583 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:42:45.449679 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.449685 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.449895 (+ 210us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.450091 (+ 196us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:45.450752 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.452752 (+ 2000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.455936 (+ 3184us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.457689 (+ 1753us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.457798 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.458027 (+ 229us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.458602 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.458815 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":954,"thread_start_us":751,"threads_started":1}
I20260810 11:42:45.459628 672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.447788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:33725 (local address 127.0.0.1:54154)
0810 11:42:45.448904 (+ 1116us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.448923 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.449045 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.450111 (+ 1066us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.450116 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.450146 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:45.450538 (+ 392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.450551 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.453024 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.453034 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.455750 (+ 2716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.455765 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.458189 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.458197 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.458390 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.458429 (+ 39us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.458536 (+ 107us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.458694 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":988,"mutex_wait_us":61,"thread_start_us":776,"threads_started":1}
I20260810 11:42:45.461455 360 tablet_service.cc:1467] Processing CreateTablet for tablet 32f09a4a6c3c47519214c4985b10e199 (DEFAULT_TABLE table=test-table [id=8f97199461a64b7a9236f26d735952e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:45.461812 32631 tablet_service.cc:1467] Processing CreateTablet for tablet 32f09a4a6c3c47519214c4985b10e199 (DEFAULT_TABLE table=test-table [id=8f97199461a64b7a9236f26d735952e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:45.463085 32631 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f09a4a6c3c47519214c4985b10e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:45.463105 360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f09a4a6c3c47519214c4985b10e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:45.463601 551 tablet_service.cc:1467] Processing CreateTablet for tablet 32f09a4a6c3c47519214c4985b10e199 (DEFAULT_TABLE table=test-table [id=8f97199461a64b7a9236f26d735952e5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:45.464964 551 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 32f09a4a6c3c47519214c4985b10e199. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:45.476426 674 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Bootstrap starting.
I20260810 11:42:45.480198 675 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Bootstrap starting.
I20260810 11:42:45.480360 674 tablet_bootstrap.cc:654] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:45.481803 674 log.cc:824] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:45.481938 676 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Bootstrap starting.
I20260810 11:42:45.484619 675 tablet_bootstrap.cc:654] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:45.485761 676 tablet_bootstrap.cc:654] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:45.486080 675 log.cc:824] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:45.487006 676 log.cc:824] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:45.488642 674 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: No bootstrap required, opened a new log
I20260810 11:42:45.488941 674 ts_tablet_manager.cc:1397] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.004s
I20260810 11:42:45.490497 675 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: No bootstrap required, opened a new log
I20260810 11:42:45.490756 675 ts_tablet_manager.cc:1397] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260810 11:42:45.497164 676 tablet_bootstrap.cc:492] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: No bootstrap required, opened a new log
I20260810 11:42:45.497447 676 ts_tablet_manager.cc:1397] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260810 11:42:45.500761 674 raft_consensus.cc:348] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.501437 674 raft_consensus.cc:374] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:45.501642 674 raft_consensus.cc:729] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dfdd78ca0402451799c80e188b9c482b, State: Initialized, Role: FOLLOWER
I20260810 11:42:45.502050 675 raft_consensus.cc:348] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.502468 675 raft_consensus.cc:374] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:45.502581 675 raft_consensus.cc:729] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 68ff5958f9aa4e14979470873c47a904, State: Initialized, Role: FOLLOWER
I20260810 11:42:45.502607 674 consensus_queue.cc:260] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.503276 675 consensus_queue.cc:260] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.505218 674 ts_tablet_manager.cc:1428] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260810 11:42:45.506402 454 heartbeater.cc:503] Master 127.3.240.254:42579 was elected leader, sending a full tablet report...
I20260810 11:42:45.507297 675 ts_tablet_manager.cc:1428] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Time spent starting tablet: real 0.016s user 0.017s sys 0.001s
I20260810 11:42:45.510844 676 raft_consensus.cc:348] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.511354 676 raft_consensus.cc:374] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:45.511467 676 raft_consensus.cc:729] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a38f775ee66e45e8a1d0781b794ad480, State: Initialized, Role: FOLLOWER
I20260810 11:42:45.512817 676 consensus_queue.cc:260] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.514370 648 heartbeater.cc:503] Master 127.3.240.254:42579 was elected leader, sending a full tablet report...
I20260810 11:42:45.515030 676 ts_tablet_manager.cc:1428] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Time spent starting tablet: real 0.017s user 0.012s sys 0.006s
I20260810 11:42:45.596697 685 raft_consensus.cc:482] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:45.597069 685 raft_consensus.cc:504] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.598842 685 leader_election.cc:288] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 68ff5958f9aa4e14979470873c47a904 (127.3.240.194:37623), dfdd78ca0402451799c80e188b9c482b (127.3.240.193:36299)
I20260810 11:42:45.607237 654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.599427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:36299 (local address 127.3.240.195:36607)
0810 11:42:45.599803 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.599824 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.599958 (+ 134us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.600520 (+ 562us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.600526 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.600607 (+ 81us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:45.600847 (+ 240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.600860 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.602484 (+ 1624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.602488 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.604626 (+ 2138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.604641 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.606380 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.606385 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.606573 (+ 188us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.606618 (+ 45us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.606749 (+ 131us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.606891 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":223,"spinlock_wait_cycles":22912}
I20260810 11:42:45.607290 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.599678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:36607 (local address 127.3.240.193:36299)
0810 11:42:45.599896 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.599911 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.599931 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:42:45.600226 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.600233 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.600337 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.600467 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:45.601029 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.602311 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.604784 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.606167 (+ 1383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.606370 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.606461 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.606796 (+ 335us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.606896 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20260810 11:42:45.607523 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.599262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:55249 (local address 127.3.240.194:37623)
0810 11:42:45.599969 (+ 707us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.599978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.600544 (+ 566us) server_negotiation.cc:407] Connection header received
0810 11:42:45.600712 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.600716 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.600779 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.600909 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:45.601466 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.602617 (+ 1151us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.605025 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.606462 (+ 1437us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.606600 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.606649 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.607138 (+ 489us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.607263 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":520}
I20260810 11:42:45.607577 686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.599569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:37623 (local address 127.3.240.195:55249)
0810 11:42:45.600438 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:45.600456 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:45.600569 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:45.600895 (+ 326us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:45.600909 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:45.600939 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:45.601266 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.601273 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.602795 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.602801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.604879 (+ 2078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:45.604895 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.606682 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.606690 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:45.606861 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.606889 (+ 28us) client_negotiation.cc:769] Sending connection context
0810 11:42:45.606998 (+ 109us) client_negotiation.cc:240] Negotiation successful
0810 11:42:45.607121 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":732,"mutex_wait_us":1,"thread_start_us":599,"threads_started":1}
I20260810 11:42:45.608731 385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f09a4a6c3c47519214c4985b10e199" candidate_uuid: "a38f775ee66e45e8a1d0781b794ad480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ff5958f9aa4e14979470873c47a904" is_pre_election: true
I20260810 11:42:45.608740 32662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f09a4a6c3c47519214c4985b10e199" candidate_uuid: "a38f775ee66e45e8a1d0781b794ad480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdd78ca0402451799c80e188b9c482b" is_pre_election: true
I20260810 11:42:45.609242 385 raft_consensus.cc:2454] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38f775ee66e45e8a1d0781b794ad480 in term 0.
I20260810 11:42:45.609263 32662 raft_consensus.cc:2454] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a38f775ee66e45e8a1d0781b794ad480 in term 0.
I20260810 11:42:45.609942 488 leader_election.cc:302] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 68ff5958f9aa4e14979470873c47a904, a38f775ee66e45e8a1d0781b794ad480, dfdd78ca0402451799c80e188b9c482b; no voters:
I20260810 11:42:45.610370 685 raft_consensus.cc:2781] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:45.610529 685 raft_consensus.cc:482] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:45.610639 685 raft_consensus.cc:3037] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:45.612900 685 raft_consensus.cc:504] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.613555 685 leader_election.cc:288] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [CANDIDATE]: Term 1 election: Requested vote from peers 68ff5958f9aa4e14979470873c47a904 (127.3.240.194:37623), dfdd78ca0402451799c80e188b9c482b (127.3.240.193:36299)
I20260810 11:42:45.613929 385 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f09a4a6c3c47519214c4985b10e199" candidate_uuid: "a38f775ee66e45e8a1d0781b794ad480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "68ff5958f9aa4e14979470873c47a904"
I20260810 11:42:45.614126 385 raft_consensus.cc:3037] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 0 FOLLOWER]: Advancing to term 1
W20260810 11:42:45.614023 651 tablet.cc:2369] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:45.614149 32662 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "32f09a4a6c3c47519214c4985b10e199" candidate_uuid: "a38f775ee66e45e8a1d0781b794ad480" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dfdd78ca0402451799c80e188b9c482b"
I20260810 11:42:45.614370 32662 raft_consensus.cc:3037] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 0 FOLLOWER]: Advancing to term 1
W20260810 11:42:45.616377 32724 tablet.cc:2369] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:45.617189 385 raft_consensus.cc:2454] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a38f775ee66e45e8a1d0781b794ad480 in term 1.
I20260810 11:42:45.617292 32662 raft_consensus.cc:2454] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a38f775ee66e45e8a1d0781b794ad480 in term 1.
I20260810 11:42:45.617834 488 leader_election.cc:302] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 68ff5958f9aa4e14979470873c47a904, a38f775ee66e45e8a1d0781b794ad480; no voters:
I20260810 11:42:45.618186 685 raft_consensus.cc:2781] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:45.618709 685 raft_consensus.cc:686] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 1 LEADER]: Becoming Leader. State: Replica: a38f775ee66e45e8a1d0781b794ad480, State: Running, Role: LEADER
I20260810 11:42:45.619323 685 consensus_queue.cc:237] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } }
I20260810 11:42:45.624755 32460 catalog_manager.cc:5495] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 reported cstate change: term changed from 0 to 1, leader changed from <none> to a38f775ee66e45e8a1d0781b794ad480 (127.3.240.195). New cstate: current_term: 1 leader_uuid: "a38f775ee66e45e8a1d0781b794ad480" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a38f775ee66e45e8a1d0781b794ad480" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 33725 } health_report { overall_health: HEALTHY } } }
I20260810 11:42:45.656805 673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.650731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54162 (local address 127.3.240.195:33725)
0810 11:42:45.650921 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.650972 (+ 51us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.650997 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:42:45.651266 (+ 269us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.651270 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.651365 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.651515 (+ 150us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:45.652068 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.652939 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.654465 (+ 1526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.654995 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.655121 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.655696 (+ 575us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:45.656220 (+ 524us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:45.656362 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.656509 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.656589 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260810 11:42:45.666332 385 raft_consensus.cc:1261] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 1 FOLLOWER]: Refusing update from remote peer a38f775ee66e45e8a1d0781b794ad480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:45.666332 32662 raft_consensus.cc:1261] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 1 FOLLOWER]: Refusing update from remote peer a38f775ee66e45e8a1d0781b794ad480: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:45.667582 687 consensus_queue.cc:1035] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "68ff5958f9aa4e14979470873c47a904" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 37623 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:45.668012 685 consensus_queue.cc:1035] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dfdd78ca0402451799c80e188b9c482b" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 36299 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:45.681571 690 mvcc.cc:205] Tried to move back new op lower bound from 7316939430555443200 to 7316939430381060096. Current Snapshot: MvccSnapshot[applied={T|T < 7316939430555443200}]
I20260810 11:42:45.683990 692 mvcc.cc:205] Tried to move back new op lower bound from 7316939430555443200 to 7316939430381060096. Current Snapshot: MvccSnapshot[applied={T|T < 7316939430555443200}]
I20260810 11:42:45.700222 32458 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:45.700671 32458 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53918:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:45.706013 32458 catalog_manager.cc:5782] T 00000000000000000000000000000000 P febd13dff2e64f6b918ffb54c4253056: Sending DeleteTablet for 3 replicas of tablet 32f09a4a6c3c47519214c4985b10e199
I20260810 11:42:45.707485 360 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f09a4a6c3c47519214c4985b10e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:41514
I20260810 11:42:45.707615 32631 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f09a4a6c3c47519214c4985b10e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:46896
I20260810 11:42:45.709870 551 tablet_service.cc:1514] Processing DeleteTablet for tablet 32f09a4a6c3c47519214c4985b10e199 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:45 UTC) from {username='kudu', principal='kudu/127.3.240.254@KRBTEST.COM'} at 127.0.0.1:54154
I20260810 11:42:45.715629 707 tablet_replica.cc:330] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: stopping tablet replica
I20260810 11:42:45.716152 707 raft_consensus.cc:2229] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:45.718739 705 tablet_replica.cc:330] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: stopping tablet replica
I20260810 11:42:45.719321 705 raft_consensus.cc:2229] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:45.719758 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 32541
I20260810 11:42:45.719976 705 raft_consensus.cc:2258] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:45.721951 705 ts_tablet_manager.cc:1905] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:45.723627 706 tablet_replica.cc:330] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: stopping tablet replica
I20260810 11:42:45.724064 706 raft_consensus.cc:2229] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:45.724599 706 raft_consensus.cc:2258] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:45.725961 707 raft_consensus.cc:2258] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:45.728188 707 ts_tablet_manager.cc:1905] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:45.729565 706 ts_tablet_manager.cc:1905] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:45.730020 705 ts_tablet_manager.cc:1918] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:45.730242 705 log.cc:1196] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/32f09a4a6c3c47519214c4985b10e199
I20260810 11:42:45.731043 705 ts_tablet_manager.cc:1939] T 32f09a4a6c3c47519214c4985b10e199 P 68ff5958f9aa4e14979470873c47a904: Deleting consensus metadata
I20260810 11:42:45.733039 32434 catalog_manager.cc:4841] TS 68ff5958f9aa4e14979470873c47a904 (127.3.240.194:37623): tablet 32f09a4a6c3c47519214c4985b10e199 (table test-table [id=8f97199461a64b7a9236f26d735952e5]) successfully deleted
I20260810 11:42:45.735324 707 ts_tablet_manager.cc:1918] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:45.735580 707 log.cc:1196] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/32f09a4a6c3c47519214c4985b10e199
I20260810 11:42:45.736258 707 ts_tablet_manager.cc:1939] T 32f09a4a6c3c47519214c4985b10e199 P a38f775ee66e45e8a1d0781b794ad480: Deleting consensus metadata
I20260810 11:42:45.736582 706 ts_tablet_manager.cc:1918] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:45.736802 706 log.cc:1196] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786361809851056-4035-0/minicluster-data/ts-0/wal/wals/32f09a4a6c3c47519214c4985b10e199
I20260810 11:42:45.737771 706 ts_tablet_manager.cc:1939] T 32f09a4a6c3c47519214c4985b10e199 P dfdd78ca0402451799c80e188b9c482b: Deleting consensus metadata
I20260810 11:42:45.739060 32436 catalog_manager.cc:4841] TS a38f775ee66e45e8a1d0781b794ad480 (127.3.240.195:33725): tablet 32f09a4a6c3c47519214c4985b10e199 (table test-table [id=8f97199461a64b7a9236f26d735952e5]) successfully deleted
I20260810 11:42:45.739713 32438 catalog_manager.cc:4841] TS dfdd78ca0402451799c80e188b9c482b (127.3.240.193:36299): tablet 32f09a4a6c3c47519214c4985b10e199 (table test-table [id=8f97199461a64b7a9236f26d735952e5]) successfully deleted
Aug 10 11:42:45 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.193@KRBTEST.COM
I20260810 11:42:45.751356 671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.721480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46910 (local address 127.3.240.193:36299)
0810 11:42:45.721660 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.721666 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.721688 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:42:45.721954 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.721962 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.722071 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.722163 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:45.724330 (+ 2167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.725642 (+ 1312us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.735109 (+ 9467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.735848 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.735991 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.743531 (+ 7540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:45.743550 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:45.743569 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:45.743620 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:45.747949 (+ 4329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.748781 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.748790 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.748794 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.748967 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.749518 (+ 551us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.749524 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.749527 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.750009 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:45.750327 (+ 318us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.750674 (+ 347us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.750906 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:42:45.752162 32714 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:45.896250 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 32541
I20260810 11:42:45.933277 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 32730
Aug 10 11:42:45 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.194@KRBTEST.COM
I20260810 11:42:45.951733 670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:45.934235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41524 (local address 127.3.240.194:37623)
0810 11:42:45.934470 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:45.934476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:45.934496 (+ 20us) server_negotiation.cc:407] Connection header received
0810 11:42:45.934601 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:45.934606 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:45.934697 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:45.934818 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:45.936186 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.936951 (+ 765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.938243 (+ 1292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:45.938630 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:45.938771 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:45.944262 (+ 5491us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:45.944278 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:45.944294 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:45.944329 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:45.947905 (+ 3576us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.949049 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.949058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.949061 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.949231 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:45.949659 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:45.949665 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:45.949668 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:45.950185 (+ 517us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:45.950558 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:45.950916 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 11:42:45.951341 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:42:45.952504 444 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:46.054198 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 32730
I20260810 11:42:46.092342 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 464
Aug 10 11:42:46 dist-test-slave-drl0 krb5kdc[32388](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362162, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.195@KRBTEST.COM
I20260810 11:42:46.112354 673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:46.093355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54164 (local address 127.3.240.195:33725)
0810 11:42:46.093604 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:46.093610 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:46.093632 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:42:46.093797 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:46.093806 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:46.093880 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:46.094000 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:46.096050 (+ 2050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:46.096930 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:46.098226 (+ 1296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:46.098689 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:46.098843 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:46.105549 (+ 6706us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:46.105563 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:46.105581 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:46.105626 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:46.109180 (+ 3554us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:46.109959 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:46.109966 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:46.109968 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:46.110122 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:46.110528 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:46.110534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:46.110537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:46.110877 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:46.111371 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:46.111713 (+ 342us) server_negotiation.cc:299] Negotiation successful
0810 11:42:46.111960 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260810 11:42:46.113315 637 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:46.254493 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 464
I20260810 11:42:46.288337 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 32413
I20260810 11:42:46.289726 32498 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:46.415513 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 32413
2026-08-10T11:42:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5306 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-10T11:42:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:46Z Disabled control of system clock
I20260810 11:42:46.533147 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35611
--webserver_interface=127.3.240.254
--webserver_port=0
--builtin_ntp_servers=127.3.240.212:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:35611
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:46.927120 719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:46.927436 719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:46.927522 719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:46.941141 719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:46.941256 719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:46.941296 719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:46.941349 719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:46.941396 719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:46.951086 719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:35611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:35611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:46.952433 719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:46.954351 719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:46.966910 726 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:46.966910 727 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:46.966992 729 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:46.967257 719 server_base.cc:1034] running on GCE node
I20260810 11:42:46.968967 719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:46.970747 719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:46.972137 719 hybrid_clock.cc:648] HybridClock initialized: now 1786362166972056 us; error 62 us; skew 500 ppm
I20260810 11:42:46.972553 719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:46.975315 719 webserver.cc:460] Webserver started at http://127.3.240.254:44169/ using document root <none> and password file <none>
I20260810 11:42:46.976030 719 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:46.976162 719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:46.976521 719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:46.979548 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "8d87dff0761c437e9f13158290eff091"
format_stamp: "Formatted at 2026-08-10 11:42:46 on dist-test-slave-drl0"
server_key: "6972b27a3798f3b70c78d7c1a55ef50b"
server_key_iv: "9f9161b13209a29d97eaeea442da3e3f"
server_key_version: "encryptionkey@0"
I20260810 11:42:46.981261 719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "8d87dff0761c437e9f13158290eff091"
format_stamp: "Formatted at 2026-08-10 11:42:46 on dist-test-slave-drl0"
server_key: "6972b27a3798f3b70c78d7c1a55ef50b"
server_key_iv: "9f9161b13209a29d97eaeea442da3e3f"
server_key_version: "encryptionkey@0"
I20260810 11:42:46.987145 719 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260810 11:42:46.990743 738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:46.992856 719 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:42:46.993093 719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "8d87dff0761c437e9f13158290eff091"
format_stamp: "Formatted at 2026-08-10 11:42:46 on dist-test-slave-drl0"
server_key: "6972b27a3798f3b70c78d7c1a55ef50b"
server_key_iv: "9f9161b13209a29d97eaeea442da3e3f"
server_key_version: "encryptionkey@0"
I20260810 11:42:46.993330 719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:47.015573 719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:47.016569 719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:47.016898 719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:47.038535 719 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:35611
I20260810 11:42:47.038582 816 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:35611 every 8 connection(s)
I20260810 11:42:47.040109 719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:47.042193 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 719
I20260810 11:42:47.042408 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:47.043100 4035 external_mini_cluster.cc:1428] Setting key 435898501db2d99d2652fdeb8f74df21
I20260810 11:42:47.045825 817 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:47.057317 822 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:47.045273 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47938 (local address 127.3.240.254:35611)
0810 11:42:47.046770 (+ 1497us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:47.046782 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:47.046825 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:42:47.048550 (+ 1725us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:47.048570 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:47.049024 (+ 454us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:47.049459 (+ 435us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:47.050015 (+ 556us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.051465 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.052752 (+ 1287us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.053402 (+ 650us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.053609 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:47.055032 (+ 1423us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:47.055042 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:47.055054 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:47.055643 (+ 589us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:47.055724 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:47.055756 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:47.055886 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:47.056127 (+ 241us) server_negotiation.cc:299] Negotiation successful
0810 11:42:47.056420 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1065,"thread_start_us":473,"threads_started":1}
I20260810 11:42:47.057785 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: Bootstrap starting.
I20260810 11:42:47.061316 817 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:47.062338 817 log.cc:824] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:47.065294 817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: No bootstrap required, opened a new log
I20260810 11:42:47.072983 817 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } }
I20260810 11:42:47.073318 817 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:47.073436 817 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8d87dff0761c437e9f13158290eff091, State: Initialized, Role: FOLLOWER
I20260810 11:42:47.073995 817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } }
I20260810 11:42:47.074201 817 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:47.074326 817 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:47.074513 817 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:47.076854 817 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } }
I20260810 11:42:47.077471 817 leader_election.cc:302] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8d87dff0761c437e9f13158290eff091; no voters:
I20260810 11:42:47.078096 817 leader_election.cc:288] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:47.078336 825 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:47.079276 825 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [term 1 LEADER]: Becoming Leader. State: Replica: 8d87dff0761c437e9f13158290eff091, State: Running, Role: LEADER
I20260810 11:42:47.079841 825 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } }
I20260810 11:42:47.080663 817 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:47.082816 827 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8d87dff0761c437e9f13158290eff091. Latest consensus state: current_term: 1 leader_uuid: "8d87dff0761c437e9f13158290eff091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } } }
I20260810 11:42:47.083316 827 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:47.082818 826 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8d87dff0761c437e9f13158290eff091" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8d87dff0761c437e9f13158290eff091" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 35611 } } }
I20260810 11:42:47.083873 826 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:47.086453 831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:47.091866 831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:47.102483 831 catalog_manager.cc:1347] Generated new cluster ID: 12a73e89c42442bf957e172c41ee427e
I20260810 11:42:47.102778 831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:47.131307 831 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:47.132654 831 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:47.148223 831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: Generated new TSK 0
I20260810 11:42:47.149494 831 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:47.160421 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.193:0
--local_ip_for_outbound_sockets=127.3.240.193
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--builtin_ntp_servers=127.3.240.212:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260810 11:42:47.556032 850 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:47.556362 850 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:47.556521 850 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:47.564196 850 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:47.564510 850 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:47.564924 850 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.193
I20260810 11:42:47.574263 850 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.193:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.3.240.193
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.193
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:47.575665 850 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:47.577476 850 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:47.591418 856 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:47.591336 860 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:47.591372 857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:47.592308 850 server_base.cc:1034] running on GCE node
I20260810 11:42:47.592973 850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:47.593968 850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:47.595235 850 hybrid_clock.cc:648] HybridClock initialized: now 1786362167595113 us; error 126 us; skew 500 ppm
I20260810 11:42:47.595606 850 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:47.599195 850 webserver.cc:460] Webserver started at http://127.3.240.193:46445/ using document root <none> and password file <none>
I20260810 11:42:47.599956 850 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:47.600090 850 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:47.600435 850 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:47.603914 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data/instance:
uuid: "15e0d40479fd4a1f803c2b01a0d9477a"
format_stamp: "Formatted at 2026-08-10 11:42:47 on dist-test-slave-drl0"
server_key: "25ff792285207fa2e547dc65d5bfcca6"
server_key_iv: "275ad429b13263d9b90cd0e484e62b65"
server_key_version: "encryptionkey@0"
I20260810 11:42:47.605315 850 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance:
uuid: "15e0d40479fd4a1f803c2b01a0d9477a"
format_stamp: "Formatted at 2026-08-10 11:42:47 on dist-test-slave-drl0"
server_key: "25ff792285207fa2e547dc65d5bfcca6"
server_key_iv: "275ad429b13263d9b90cd0e484e62b65"
server_key_version: "encryptionkey@0"
I20260810 11:42:47.611871 850 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260810 11:42:47.615850 867 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:47.617799 850 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:42:47.618088 850 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal
uuid: "15e0d40479fd4a1f803c2b01a0d9477a"
format_stamp: "Formatted at 2026-08-10 11:42:47 on dist-test-slave-drl0"
server_key: "25ff792285207fa2e547dc65d5bfcca6"
server_key_iv: "275ad429b13263d9b90cd0e484e62b65"
server_key_version: "encryptionkey@0"
I20260810 11:42:47.618413 850 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:47.653586 850 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:47.654482 850 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:47.654809 850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:47.655855 850 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:47.657562 850 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:47.657665 850 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:47.657809 850 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:47.657910 850 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:47.697860 850 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.193:39743
I20260810 11:42:47.697899 1041 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.193:39743 every 8 connection(s)
I20260810 11:42:47.699466 850 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/data/info.pb
I20260810 11:42:47.702013 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 850
I20260810 11:42:47.702164 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-0/wal/instance
I20260810 11:42:47.702761 4035 external_mini_cluster.cc:1428] Setting key 0fd55308af0a5588cf6df64fff95e68c
I20260810 11:42:47.706447 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.194:0
--local_ip_for_outbound_sockets=127.3.240.194
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--builtin_ntp_servers=127.3.240.212:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:47.719172 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:47.703521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.193:52707 (local address 127.3.240.254:35611)
0810 11:42:47.704099 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:47.704104 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:47.705772 (+ 1668us) server_negotiation.cc:407] Connection header received
0810 11:42:47.709521 (+ 3749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:47.709525 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:47.709591 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:47.709689 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:47.711349 (+ 1660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.712189 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.715501 (+ 3312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.715905 (+ 404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.716020 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:47.717991 (+ 1971us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:47.717996 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:47.717999 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:47.718018 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:47.718055 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:47.718067 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:47.718138 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:47.718477 (+ 339us) server_negotiation.cc:299] Negotiation successful
0810 11:42:47.718557 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":317,"threads_started":1}
I20260810 11:42:47.720234 1048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:47.704288 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35611 (local address 127.3.240.193:52707)
0810 11:42:47.705598 (+ 1310us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:47.705645 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:47.709203 (+ 3558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:47.710037 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:47.710051 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:47.710166 (+ 115us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:47.711139 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:47.711170 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.712356 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.712364 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:47.715148 (+ 2784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:47.715174 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:47.716153 (+ 979us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:47.716163 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:47.716521 (+ 358us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:47.717575 (+ 1054us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:47.717581 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:47.717635 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:47.717737 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:47.718168 (+ 431us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:47.718182 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:47.718193 (+ 11us) client_negotiation.cc:769] Sending connection context
0810 11:42:47.718385 (+ 192us) client_negotiation.cc:240] Negotiation successful
0810 11:42:47.718799 (+ 414us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":382,"threads_started":1}
I20260810 11:42:47.722497 1042 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35611
I20260810 11:42:47.723094 1042 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:47.724097 1042 heartbeater.cc:511] Master 127.3.240.254:35611 requested a full tablet report, sending...
I20260810 11:42:47.727066 769 ts_manager.cc:194] Registered new tserver with Master: 15e0d40479fd4a1f803c2b01a0d9477a (127.3.240.193:39743)
I20260810 11:42:47.728849 769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.193:52707
W20260810 11:42:48.083292 1049 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:48.083603 1049 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:48.083763 1049 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:48.092442 1049 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:48.092598 1049 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:48.092794 1049 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.194
I20260810 11:42:48.102763 1049 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.194:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.3.240.194
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.194
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:48.104104 1049 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:48.106024 1049 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:48.120721 1057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:48.120808 1068 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:48.120760 1063 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:48.121630 1049 server_base.cc:1034] running on GCE node
I20260810 11:42:48.122215 1049 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:48.123271 1049 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:48.124527 1049 hybrid_clock.cc:648] HybridClock initialized: now 1786362168124478 us; error 41 us; skew 500 ppm
I20260810 11:42:48.124871 1049 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:48.127384 1049 webserver.cc:460] Webserver started at http://127.3.240.194:43525/ using document root <none> and password file <none>
I20260810 11:42:48.128590 1049 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:48.128866 1049 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:48.129235 1049 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:48.132172 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data/instance:
uuid: "cffc0ee0736548ac937dd7b8430f51e2"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "517711c9c50ba7b2a416578acc2fb3f2"
server_key_iv: "9ec1afdaf8b02934dbefde6296c77371"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.132984 1049 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance:
uuid: "cffc0ee0736548ac937dd7b8430f51e2"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "517711c9c50ba7b2a416578acc2fb3f2"
server_key_iv: "9ec1afdaf8b02934dbefde6296c77371"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.138053 1049 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260810 11:42:48.141769 1075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.143942 1049 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260810 11:42:48.144236 1049 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal
uuid: "cffc0ee0736548ac937dd7b8430f51e2"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "517711c9c50ba7b2a416578acc2fb3f2"
server_key_iv: "9ec1afdaf8b02934dbefde6296c77371"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.144537 1049 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:48.165586 1049 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:48.166468 1049 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:48.166785 1049 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:48.167912 1049 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:48.169518 1049 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:48.169656 1049 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.169806 1049 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:48.169904 1049 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.209759 1049 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.194:35259
I20260810 11:42:48.209810 1231 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.194:35259 every 8 connection(s)
I20260810 11:42:48.211485 1049 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/data/info.pb
I20260810 11:42:48.218423 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 1049
I20260810 11:42:48.218624 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal/instance
I20260810 11:42:48.219252 4035 external_mini_cluster.cc:1428] Setting key 7b5d3be3ef218d988e3c7da0e60599d8
I20260810 11:42:48.222801 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.3.240.195:0
--local_ip_for_outbound_sockets=127.3.240.195
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--builtin_ntp_servers=127.3.240.212:38685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260810 11:42:48.230561 1047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.215161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.194:50863 (local address 127.3.240.254:35611)
0810 11:42:48.215405 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.215410 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.216911 (+ 1501us) server_negotiation.cc:407] Connection header received
0810 11:42:48.219163 (+ 2252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.219168 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.219243 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.219390 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:48.220731 (+ 1341us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.221790 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.224334 (+ 2544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.224933 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.225041 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.229300 (+ 4259us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:48.229305 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:48.229308 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:48.229327 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:48.229500 (+ 173us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:48.229559 (+ 59us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:48.229729 (+ 170us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.230090 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.230175 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260810 11:42:48.232028 1236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.215788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35611 (local address 127.3.240.194:50863)
0810 11:42:48.216772 (+ 984us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.216809 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.218780 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.219643 (+ 863us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.219656 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.219767 (+ 111us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:48.220551 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.220572 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.221973 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.221979 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.224141 (+ 2162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.224162 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.225178 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.225188 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.225613 (+ 425us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.228876 (+ 3263us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:48.228883 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:48.228942 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:48.229068 (+ 126us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:48.229756 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:48.229771 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:48.229783 (+ 12us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.229999 (+ 216us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.230469 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":323,"threads_started":1}
I20260810 11:42:48.233661 1233 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35611
I20260810 11:42:48.233982 1233 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:48.234911 1233 heartbeater.cc:511] Master 127.3.240.254:35611 requested a full tablet report, sending...
I20260810 11:42:48.236901 769 ts_manager.cc:194] Registered new tserver with Master: cffc0ee0736548ac937dd7b8430f51e2 (127.3.240.194:35259)
I20260810 11:42:48.238096 769 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.194:50863
W20260810 11:42:48.610723 1237 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:48.611145 1237 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:48.611447 1237 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:48.619993 1237 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:48.620098 1237 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:48.620285 1237 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.3.240.195
I20260810 11:42:48.631477 1237 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:38685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.3.240.195:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.3.240.195
--webserver_port=0
--tserver_master_addrs=127.3.240.254:35611
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.3.240.195
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:48.632860 1237 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:48.634688 1237 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:48.649516 1247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:48.649544 1248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:48.649785 1250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:48.650695 1237 server_base.cc:1034] running on GCE node
I20260810 11:42:48.651556 1237 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:48.652827 1237 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:48.654407 1237 hybrid_clock.cc:648] HybridClock initialized: now 1786362168654305 us; error 51 us; skew 500 ppm
I20260810 11:42:48.655025 1237 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260810 11:42:48.658005 1237 webserver.cc:460] Webserver started at http://127.3.240.195:40065/ using document root <none> and password file <none>
I20260810 11:42:48.658907 1237 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:48.659132 1237 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:48.659482 1237 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:48.662501 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data/instance:
uuid: "0a0ea46111334037ad18d669587bae6d"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "02f53d7706ed2a23eb3286d69a242249"
server_key_iv: "49256f5b0083c6f1c5a3ecbc07e71b1c"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.663363 1237 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance:
uuid: "0a0ea46111334037ad18d669587bae6d"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "02f53d7706ed2a23eb3286d69a242249"
server_key_iv: "49256f5b0083c6f1c5a3ecbc07e71b1c"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.668901 1237 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260810 11:42:48.673336 1259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.675127 1237 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260810 11:42:48.675330 1237 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal
uuid: "0a0ea46111334037ad18d669587bae6d"
format_stamp: "Formatted at 2026-08-10 11:42:48 on dist-test-slave-drl0"
server_key: "02f53d7706ed2a23eb3286d69a242249"
server_key_iv: "49256f5b0083c6f1c5a3ecbc07e71b1c"
server_key_version: "encryptionkey@0"
I20260810 11:42:48.675570 1237 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:48.691465 1237 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:48.692297 1237 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:48.692616 1237 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:48.693733 1237 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260810 11:42:48.695554 1237 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260810 11:42:48.695698 1237 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.695837 1237 ts_tablet_manager.cc:610] Registered 0 tablets
I20260810 11:42:48.695928 1237 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:48.732714 1042 heartbeater.cc:503] Master 127.3.240.254:35611 was elected leader, sending a full tablet report...
I20260810 11:42:48.739620 1237 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.195:43329
I20260810 11:42:48.739846 1417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.195:43329 every 8 connection(s)
I20260810 11:42:48.741046 1237 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/data/info.pb
I20260810 11:42:48.743074 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 1237
I20260810 11:42:48.743260 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal/instance
I20260810 11:42:48.743703 4035 external_mini_cluster.cc:1428] Setting key 28df175d2cc70009c118acfcb00e0863
I20260810 11:42:48.755913 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.744834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:48719 (local address 127.3.240.254:35611)
0810 11:42:48.745401 (+ 567us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.745408 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.747077 (+ 1669us) server_negotiation.cc:407] Connection header received
0810 11:42:48.749339 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.749343 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.749412 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.749511 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:48.750699 (+ 1188us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.751555 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.753132 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.753534 (+ 402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.753652 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.755156 (+ 1504us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:48.755161 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:48.755164 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:48.755181 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:48.755220 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:48.755233 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:48.755319 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.755553 (+ 234us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.755629 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":442,"thread_start_us":330,"threads_started":1}
I20260810 11:42:48.756979 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.745756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:35611 (local address 127.3.240.195:48719)
0810 11:42:48.746886 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.746960 (+ 74us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.749050 (+ 2090us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.749829 (+ 779us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.749843 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.749938 (+ 95us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:42:48.750510 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.750527 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.751733 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.751738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.752979 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.752992 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.753790 (+ 798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.753795 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.754006 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.754735 (+ 729us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0810 11:42:48.754739 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:42:48.754780 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0810 11:42:48.754860 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:42:48.755342 (+ 482us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:42:48.755351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:42:48.755358 (+ 7us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.755524 (+ 166us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.755818 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":375,"threads_started":1}
I20260810 11:42:48.758404 1418 heartbeater.cc:347] Connected to a master server at 127.3.240.254:35611
I20260810 11:42:48.758764 1418 heartbeater.cc:464] Registering TS with master...
I20260810 11:42:48.760135 1418 heartbeater.cc:511] Master 127.3.240.254:35611 requested a full tablet report, sending...
I20260810 11:42:48.761971 767 ts_manager.cc:194] Registered new tserver with Master: 0a0ea46111334037ad18d669587bae6d (127.3.240.195:43329)
I20260810 11:42:48.762904 767 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.3.240.195:48719
I20260810 11:42:48.772197 4035 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260810 11:42:48.784253 1420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.776985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47948 (local address 127.3.240.254:35611)
0810 11:42:48.777224 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.777234 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.777577 (+ 343us) server_negotiation.cc:407] Connection header received
0810 11:42:48.778031 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.778035 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.778097 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.778188 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:48.778765 (+ 577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.779559 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.781523 (+ 1964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.782044 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.782145 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.783336 (+ 1191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:48.783341 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:48.783344 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:48.783360 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:48.783400 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:48.783413 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:48.783491 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.783751 (+ 260us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.783832 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 11:42:48.789275 767 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47948:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260810 11:42:48.796931 767 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260810 11:42:48.842238 1434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.812063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44910)
0810 11:42:48.826572 (+ 14509us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.826645 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.827249 (+ 604us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.832921 (+ 5672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.832933 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.832975 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:48.833439 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.833455 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.836142 (+ 2687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.836150 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.838984 (+ 2834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.839013 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.841091 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.841100 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.841355 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.841426 (+ 71us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.841563 (+ 137us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.841773 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14160,"mutex_wait_us":165,"thread_start_us":15557,"threads_started":1}
I20260810 11:42:48.843050 1438 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.812741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.195:43329 (local address 127.0.0.1:47976)
0810 11:42:48.832333 (+ 19592us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.832353 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.832587 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.833728 (+ 1141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.833732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.833755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:48.834221 (+ 466us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.834240 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.836550 (+ 2310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.836558 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.839536 (+ 2978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.839558 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.842029 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.842035 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.842288 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.842335 (+ 47us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.842440 (+ 105us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.842586 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19438,"mutex_wait_us":95,"thread_start_us":19278,"threads_started":1}
I20260810 11:42:48.843856 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.812291 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:35259 (local address 127.0.0.1:60454)
0810 11:42:48.831124 (+ 18833us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.831156 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.831330 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.834007 (+ 2677us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.834014 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.834051 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:48.834590 (+ 539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.834601 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.836992 (+ 2391us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.837000 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.839625 (+ 2625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.839637 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.842831 (+ 3194us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.842840 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.843097 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.843150 (+ 53us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.843258 (+ 108us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.843429 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18660,"mutex_wait_us":1,"thread_start_us":18446,"threads_started":1}
I20260810 11:42:48.843924 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.812367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44910 (local address 127.3.240.193:39743)
0810 11:42:48.832056 (+ 19689us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.832064 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.832099 (+ 35us) server_negotiation.cc:407] Connection header received
0810 11:42:48.832205 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.832216 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.832587 (+ 371us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.833633 (+ 1046us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:48.833877 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.835835 (+ 1958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.839263 (+ 3428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.840780 (+ 1517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.842872 (+ 2092us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.843202 (+ 330us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.843289 (+ 87us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.843498 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19474,"thread_start_us":19257,"threads_started":1}
I20260810 11:42:48.844164 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.812907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60454 (local address 127.3.240.194:35259)
0810 11:42:48.833394 (+ 20487us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.833405 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.833436 (+ 31us) server_negotiation.cc:407] Connection header received
0810 11:42:48.833532 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.833539 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.833753 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.833981 (+ 228us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:48.834752 (+ 771us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.836099 (+ 1347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.839833 (+ 3734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.842203 (+ 2370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.842365 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.842640 (+ 275us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.843504 (+ 864us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.843717 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20306,"thread_start_us":21310,"threads_started":1}
I20260810 11:42:48.844508 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.813487 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47976 (local address 127.3.240.195:43329)
0810 11:42:48.830512 (+ 17025us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.830525 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.832457 (+ 1932us) server_negotiation.cc:407] Connection header received
0810 11:42:48.832942 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.832951 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.833164 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.833343 (+ 179us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:48.834574 (+ 1231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.836381 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.839791 (+ 3410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.841370 (+ 1579us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.841534 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.841880 (+ 346us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.843361 (+ 1481us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.843608 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16798,"thread_start_us":16631,"threads_started":1}
I20260810 11:42:48.848374 1326 tablet_service.cc:1467] Processing CreateTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d (DEFAULT_TABLE table=test-table [id=b690b2fadf174e7db59e1b63e96c5fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:48.848373 940 tablet_service.cc:1467] Processing CreateTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d (DEFAULT_TABLE table=test-table [id=b690b2fadf174e7db59e1b63e96c5fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:48.848604 1139 tablet_service.cc:1467] Processing CreateTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d (DEFAULT_TABLE table=test-table [id=b690b2fadf174e7db59e1b63e96c5fcf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260810 11:42:48.849903 1139 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506eb796c71e4e779ff12c1bd7fa1a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:48.849924 1326 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506eb796c71e4e779ff12c1bd7fa1a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:48.849915 940 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 506eb796c71e4e779ff12c1bd7fa1a6d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:48.862087 1444 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Bootstrap starting.
I20260810 11:42:48.865481 1445 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Bootstrap starting.
I20260810 11:42:48.866585 1446 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Bootstrap starting.
I20260810 11:42:48.867465 1444 tablet_bootstrap.cc:654] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:48.869534 1444 log.cc:824] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:48.869896 1445 tablet_bootstrap.cc:654] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:48.870249 1446 tablet_bootstrap.cc:654] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:48.871018 1445 log.cc:824] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:48.871560 1446 log.cc:824] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:48.873196 1444 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: No bootstrap required, opened a new log
I20260810 11:42:48.873481 1444 ts_tablet_manager.cc:1397] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Time spent bootstrapping tablet: real 0.012s user 0.008s sys 0.000s
I20260810 11:42:48.873894 1445 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: No bootstrap required, opened a new log
I20260810 11:42:48.874162 1445 ts_tablet_manager.cc:1397] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 11:42:48.874641 1446 tablet_bootstrap.cc:492] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: No bootstrap required, opened a new log
I20260810 11:42:48.874912 1446 ts_tablet_manager.cc:1397] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260810 11:42:48.883275 1444 raft_consensus.cc:348] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.883811 1444 raft_consensus.cc:374] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:48.883916 1444 raft_consensus.cc:729] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cffc0ee0736548ac937dd7b8430f51e2, State: Initialized, Role: FOLLOWER
I20260810 11:42:48.884796 1444 consensus_queue.cc:260] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.886318 1233 heartbeater.cc:503] Master 127.3.240.254:35611 was elected leader, sending a full tablet report...
I20260810 11:42:48.886652 1446 raft_consensus.cc:348] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.887001 1444 ts_tablet_manager.cc:1428] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20260810 11:42:48.887109 1446 raft_consensus.cc:374] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:48.887223 1446 raft_consensus.cc:729] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0a0ea46111334037ad18d669587bae6d, State: Initialized, Role: FOLLOWER
I20260810 11:42:48.885833 1445 raft_consensus.cc:348] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.887539 1445 raft_consensus.cc:374] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:48.887681 1445 raft_consensus.cc:729] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 15e0d40479fd4a1f803c2b01a0d9477a, State: Initialized, Role: FOLLOWER
I20260810 11:42:48.888357 1445 consensus_queue.cc:260] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.888379 1446 consensus_queue.cc:260] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.888775 1454 raft_consensus.cc:482] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260810 11:42:48.889194 1454 raft_consensus.cc:504] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.896795 1418 heartbeater.cc:503] Master 127.3.240.254:35611 was elected leader, sending a full tablet report...
I20260810 11:42:48.897717 1446 ts_tablet_manager.cc:1428] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Time spent starting tablet: real 0.023s user 0.011s sys 0.009s
I20260810 11:42:48.898998 1454 leader_election.cc:288] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 15e0d40479fd4a1f803c2b01a0d9477a (127.3.240.193:39743), cffc0ee0736548ac937dd7b8430f51e2 (127.3.240.194:35259)
I20260810 11:42:48.902510 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.892191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.3.240.195:59065)
0810 11:42:48.892667 (+ 476us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.892693 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.892821 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.893790 (+ 969us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.893799 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.893839 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:48.894108 (+ 269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.894123 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.896039 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.896049 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.899886 (+ 3837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.899898 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.901626 (+ 1728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.901635 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.901897 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.901935 (+ 38us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.902024 (+ 89us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.902171 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":305}
I20260810 11:42:48.902647 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.893033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:59065 (local address 127.3.240.193:39743)
0810 11:42:48.893225 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.893232 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.893251 (+ 19us) server_negotiation.cc:407] Connection header received
0810 11:42:48.893314 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.893319 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.893429 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.893540 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:48.894325 (+ 785us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.895802 (+ 1477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.900054 (+ 4252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.901282 (+ 1228us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.901453 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.901525 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.902256 (+ 731us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.902406 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:42:48.903684 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506eb796c71e4e779ff12c1bd7fa1a6d" candidate_uuid: "0a0ea46111334037ad18d669587bae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" is_pre_election: true
I20260810 11:42:48.904333 972 raft_consensus.cc:2454] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0a0ea46111334037ad18d669587bae6d in term 0.
I20260810 11:42:48.905412 1262 leader_election.cc:302] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a0ea46111334037ad18d669587bae6d, 15e0d40479fd4a1f803c2b01a0d9477a; no voters:
I20260810 11:42:48.905859 1454 raft_consensus.cc:2781] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260810 11:42:48.906132 1454 raft_consensus.cc:482] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260810 11:42:48.906337 1454 raft_consensus.cc:3037] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:48.906455 1445 ts_tablet_manager.cc:1428] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Time spent starting tablet: real 0.032s user 0.022s sys 0.009s
I20260810 11:42:48.910331 1454 raft_consensus.cc:504] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.910784 1421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.898174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.194:35259 (local address 127.3.240.195:38783)
0810 11:42:48.902882 (+ 4708us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:48.902903 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:42:48.903090 (+ 187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:42:48.903643 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:42:48.903650 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:42:48.903683 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0810 11:42:48.903887 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.903900 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.905886 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.905891 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.907955 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:42:48.907977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.909885 (+ 1908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.909893 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:42:48.910139 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.910191 (+ 52us) client_negotiation.cc:769] Sending connection context
0810 11:42:48.910310 (+ 119us) client_negotiation.cc:240] Negotiation successful
0810 11:42:48.910463 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4577,"thread_start_us":8555,"threads_started":1}
I20260810 11:42:48.911536 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.898717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.3.240.195:38783 (local address 127.3.240.194:35259)
0810 11:42:48.899194 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.899202 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.902994 (+ 3792us) server_negotiation.cc:407] Connection header received
0810 11:42:48.903260 (+ 266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.903267 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.903370 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.903440 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0810 11:42:48.904063 (+ 623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.905078 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.908245 (+ 3167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.909464 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.909683 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.909750 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.910416 (+ 666us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.911254 (+ 838us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337}
I20260810 11:42:48.911744 972 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506eb796c71e4e779ff12c1bd7fa1a6d" candidate_uuid: "0a0ea46111334037ad18d669587bae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "15e0d40479fd4a1f803c2b01a0d9477a"
I20260810 11:42:48.911959 972 raft_consensus.cc:3037] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:48.912587 1170 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506eb796c71e4e779ff12c1bd7fa1a6d" candidate_uuid: "0a0ea46111334037ad18d669587bae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffc0ee0736548ac937dd7b8430f51e2"
I20260810 11:42:48.912665 1172 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "506eb796c71e4e779ff12c1bd7fa1a6d" candidate_uuid: "0a0ea46111334037ad18d669587bae6d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cffc0ee0736548ac937dd7b8430f51e2" is_pre_election: true
I20260810 11:42:48.911561 1454 leader_election.cc:288] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [CANDIDATE]: Term 1 election: Requested vote from peers 15e0d40479fd4a1f803c2b01a0d9477a (127.3.240.193:39743), cffc0ee0736548ac937dd7b8430f51e2 (127.3.240.194:35259)
I20260810 11:42:48.912955 1170 raft_consensus.cc:3037] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:48.914724 972 raft_consensus.cc:2454] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a0ea46111334037ad18d669587bae6d in term 1.
I20260810 11:42:48.915222 1262 leader_election.cc:302] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0a0ea46111334037ad18d669587bae6d, 15e0d40479fd4a1f803c2b01a0d9477a; no voters:
I20260810 11:42:48.915560 1454 raft_consensus.cc:2781] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:48.915745 1454 raft_consensus.cc:686] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 1 LEADER]: Becoming Leader. State: Replica: 0a0ea46111334037ad18d669587bae6d, State: Running, Role: LEADER
I20260810 11:42:48.916249 1170 raft_consensus.cc:2454] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0a0ea46111334037ad18d669587bae6d in term 1.
I20260810 11:42:48.916234 1454 consensus_queue.cc:237] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } }
I20260810 11:42:48.921229 767 catalog_manager.cc:5495] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d reported cstate change: term changed from 0 to 1, leader changed from <none> to 0a0ea46111334037ad18d669587bae6d (127.3.240.195). New cstate: current_term: 1 leader_uuid: "0a0ea46111334037ad18d669587bae6d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0a0ea46111334037ad18d669587bae6d" member_type: VOTER last_known_addr { host: "127.3.240.195" port: 43329 } health_report { overall_health: HEALTHY } } }
I20260810 11:42:48.949216 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:48.941964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47990 (local address 127.3.240.195:43329)
0810 11:42:48.942219 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:48.942227 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:48.942248 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:42:48.942507 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:48.942511 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:48.942606 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:48.942776 (+ 170us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:42:48.943256 (+ 480us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.944447 (+ 1191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.946359 (+ 1912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:48.946878 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:48.947109 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:48.947752 (+ 643us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:42:48.948455 (+ 703us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:42:48.948637 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:48.948732 (+ 95us) server_negotiation.cc:299] Negotiation successful
0810 11:42:48.948836 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260810 11:42:48.952702 1044 tablet.cc:2369] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:48.958109 972 raft_consensus.cc:1261] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 1 FOLLOWER]: Refusing update from remote peer 0a0ea46111334037ad18d669587bae6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:48.958069 1170 raft_consensus.cc:1261] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 1 FOLLOWER]: Refusing update from remote peer 0a0ea46111334037ad18d669587bae6d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:42:48.959164 1460 consensus_queue.cc:1035] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "cffc0ee0736548ac937dd7b8430f51e2" member_type: VOTER last_known_addr { host: "127.3.240.194" port: 35259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:42:48.959517 1454 consensus_queue.cc:1035] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [LEADER]: Connected to new peer: Peer: permanent_uuid: "15e0d40479fd4a1f803c2b01a0d9477a" member_type: VOTER last_known_addr { host: "127.3.240.193" port: 39743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260810 11:42:48.964287 1235 tablet.cc:2369] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:42:48.996860 769 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:48.997229 769 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47948:
table { table_name: "test-table" } modify_external_catalogs: true
I20260810 11:42:49.001847 769 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 8d87dff0761c437e9f13158290eff091: Sending DeleteTablet for 3 replicas of tablet 506eb796c71e4e779ff12c1bd7fa1a6d
I20260810 11:42:49.003378 940 tablet_service.cc:1514] Processing DeleteTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:48 UTC) from {username='slave'} at 127.0.0.1:44910
I20260810 11:42:49.003523 1326 tablet_service.cc:1514] Processing DeleteTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:48 UTC) from {username='slave'} at 127.0.0.1:47976
I20260810 11:42:49.003512 1139 tablet_service.cc:1514] Processing DeleteTablet for tablet 506eb796c71e4e779ff12c1bd7fa1a6d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-10 11:42:48 UTC) from {username='slave'} at 127.0.0.1:60454
I20260810 11:42:49.006384 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 850
I20260810 11:42:49.019877 1484 tablet_replica.cc:330] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: stopping tablet replica
I20260810 11:42:49.020399 1484 raft_consensus.cc:2229] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:49.021001 1484 raft_consensus.cc:2258] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:49.024411 1484 ts_tablet_manager.cc:1905] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:49.027833 1437 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.008177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44914 (local address 127.3.240.193:39743)
0810 11:42:49.008405 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:49.008415 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:49.012040 (+ 3625us) server_negotiation.cc:407] Connection header received
0810 11:42:49.013098 (+ 1058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:49.013108 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:49.013258 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:49.013366 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:49.014774 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.016375 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.021645 (+ 5270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.022353 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.022471 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:49.025373 (+ 2902us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:49.025391 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:49.025416 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:49.026680 (+ 1264us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:49.026759 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:49.026804 (+ 45us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:49.026985 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:49.027261 (+ 276us) server_negotiation.cc:299] Negotiation successful
0810 11:42:49.027410 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260810 11:42:49.028657 1482 tablet_replica.cc:330] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a: stopping tablet replica
I20260810 11:42:49.029076 1035 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:49.029858 1483 tablet_replica.cc:330] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: stopping tablet replica
I20260810 11:42:49.030251 1483 raft_consensus.cc:2229] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 1 LEADER]: Raft consensus shutting down.
I20260810 11:42:49.031028 1483 raft_consensus.cc:2258] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:49.033281 1483 ts_tablet_manager.cc:1905] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260810 11:42:49.037118 1484 ts_tablet_manager.cc:1918] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:49.037459 1484 log.cc:1196] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-1/wal/wals/506eb796c71e4e779ff12c1bd7fa1a6d
I20260810 11:42:49.038159 1484 ts_tablet_manager.cc:1939] T 506eb796c71e4e779ff12c1bd7fa1a6d P cffc0ee0736548ac937dd7b8430f51e2: Deleting consensus metadata
I20260810 11:42:49.040133 741 catalog_manager.cc:4841] TS cffc0ee0736548ac937dd7b8430f51e2 (127.3.240.194:35259): tablet 506eb796c71e4e779ff12c1bd7fa1a6d (table test-table [id=b690b2fadf174e7db59e1b63e96c5fcf]) successfully deleted
I20260810 11:42:49.040834 1483 ts_tablet_manager.cc:1918] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260810 11:42:49.041182 1483 log.cc:1196] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Deleting WAL directory at /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786361809851056-4035-0/minicluster-data/ts-2/wal/wals/506eb796c71e4e779ff12c1bd7fa1a6d
I20260810 11:42:49.042104 1483 ts_tablet_manager.cc:1939] T 506eb796c71e4e779ff12c1bd7fa1a6d P 0a0ea46111334037ad18d669587bae6d: Deleting consensus metadata
I20260810 11:42:49.043895 740 catalog_manager.cc:4841] TS 0a0ea46111334037ad18d669587bae6d (127.3.240.195:43329): tablet 506eb796c71e4e779ff12c1bd7fa1a6d (table test-table [id=b690b2fadf174e7db59e1b63e96c5fcf]) successfully deleted
I20260810 11:42:49.029114 1482 raft_consensus.cc:2229] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260810 11:42:49.142668 1482 raft_consensus.cc:2258] T 506eb796c71e4e779ff12c1bd7fa1a6d P 15e0d40479fd4a1f803c2b01a0d9477a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260810 11:42:49.142853 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 850
W20260810 11:42:49.165169 744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.3.240.193:39743 (error 108)
I20260810 11:42:49.166244 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1049
I20260810 11:42:49.167340 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.166892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44918)
0810 11:42:49.167090 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.167202 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 11:42:49.167625 744 catalog_manager.cc:4584] TS 15e0d40479fd4a1f803c2b01a0d9477a (127.3.240.193:39743): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 506eb796c71e4e779ff12c1bd7fa1a6d: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
I20260810 11:42:49.175338 1440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.167634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60458 (local address 127.3.240.194:35259)
0810 11:42:49.167852 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:49.167859 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:49.167880 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:42:49.167950 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:49.167956 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:49.168065 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:49.168152 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:49.168825 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.169795 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.171736 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.172246 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.172516 (+ 270us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:49.173714 (+ 1198us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:49.173724 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:49.173739 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:49.174376 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:49.174432 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:49.174453 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:49.174550 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:49.174804 (+ 254us) server_negotiation.cc:299] Negotiation successful
0810 11:42:49.174995 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260810 11:42:49.176271 1227 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:49.221623 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.221123 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44926)
0810 11:42:49.221352 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.221458 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 11:42:49.222985 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.222569 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44934)
0810 11:42:49.222738 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.222821 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 11:42:49.283862 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.283040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44946)
0810 11:42:49.283398 (+ 358us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.283567 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 11:42:49.285890 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.285279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44948)
0810 11:42:49.285519 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.285674 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 11:42:49.297542 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 1049
I20260810 11:42:49.321487 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1237
I20260810 11:42:49.331349 1442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.322375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.3.240.195:43329)
0810 11:42:49.322595 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:49.322602 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:49.322625 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:42:49.322808 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:49.322814 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:49.322962 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:49.323090 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:49.323612 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.325027 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.326890 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:49.327650 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:49.327802 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:49.329253 (+ 1451us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:49.329265 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:49.329286 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:49.330184 (+ 898us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:49.330255 (+ 71us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:49.330282 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:49.330418 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:49.330635 (+ 217us) server_negotiation.cc:299] Negotiation successful
0810 11:42:49.330858 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:42:49.332572 1413 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:49.367439 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.366906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44960)
0810 11:42:49.367162 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.367263 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260810 11:42:49.368518 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.368126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44966)
0810 11:42:49.368341 (+ 215us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.368407 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 11:42:49.462646 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 1237
I20260810 11:42:49.491976 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 719
I20260810 11:42:49.493378 811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:49.621940 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:49.621152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.193:39743 (local address 127.0.0.1:44982)
0810 11:42:49.621596 (+ 444us) negotiation.cc:107] Waiting for socket to connect
0810 11:42:49.621724 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.193:39743: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":249}
I20260810 11:42:49.623652 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 719
2026-08-10T11:42:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3217 ms)
[----------] 6 tests from EncryptionPolicyTest (25360 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:49 dist-test-slave-drl0 krb5kdc[1501](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:49 dist-test-slave-drl0 krb5kdc[1501](info): set up 2 sockets
Aug 10 11:42:49 dist-test-slave-drl0 krb5kdc[1501](info): commencing operation
krb5kdc: starting...
W20260810 11:42:51.048460 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.357s user 0.000s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:51 dist-test-slave-drl0 krb5kdc[1501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:42:51.411093 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45135
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:51.799023 1519 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:51.799394 1519 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:51.799561 1519 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:51.807027 1519 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:51.807183 1519 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:51.807268 1519 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:51.807366 1519 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:51.807431 1519 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:51.817569 1519 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45135
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45135
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:51.818861 1519 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:51.820199 1519 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:51.834421 1527 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:51.834375 1528 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:51.834492 1535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:51.836306 1519 server_base.cc:1034] running on GCE node
I20260810 11:42:51.837117 1519 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:51.838660 1519 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:51.839921 1519 hybrid_clock.cc:648] HybridClock initialized: now 1786362171839845 us; error 66 us; skew 500 ppm
I20260810 11:42:51.840340 1519 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:51 dist-test-slave-drl0 krb5kdc[1501](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362171, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:51.848068 1519 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:42:51.850471 1519 webserver.cc:460] Webserver started at http://127.0.0.1:33239/ using document root <none> and password file <none>
I20260810 11:42:51.851545 1519 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:51.851724 1519 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:51.852237 1519 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:51.856360 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "d562d329b4164ac1a11cffd8e78456fd"
format_stamp: "Formatted at 2026-08-10 11:42:51 on dist-test-slave-drl0"
server_key: "b16d0a201c720ee4877b6eb91f5eb13c"
server_key_iv: "c5e4468e931eb0c4cae93ed2d830bb8e"
server_key_version: "encryptionkey@0"
I20260810 11:42:51.857569 1519 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "d562d329b4164ac1a11cffd8e78456fd"
format_stamp: "Formatted at 2026-08-10 11:42:51 on dist-test-slave-drl0"
server_key: "b16d0a201c720ee4877b6eb91f5eb13c"
server_key_iv: "c5e4468e931eb0c4cae93ed2d830bb8e"
server_key_version: "encryptionkey@0"
I20260810 11:42:51.863580 1519 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 11:42:51.867429 1545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:51.868929 1519 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:42:51.869149 1519 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "d562d329b4164ac1a11cffd8e78456fd"
format_stamp: "Formatted at 2026-08-10 11:42:51 on dist-test-slave-drl0"
server_key: "b16d0a201c720ee4877b6eb91f5eb13c"
server_key_iv: "c5e4468e931eb0c4cae93ed2d830bb8e"
server_key_version: "encryptionkey@0"
I20260810 11:42:51.869367 1519 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:51.892740 1519 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:51.897770 1519 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:51.898126 1519 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:51.920142 1519 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45135
I20260810 11:42:51.920186 1625 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45135 every 8 connection(s)
I20260810 11:42:51.921593 1519 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:51.925568 1626 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:51.928007 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 1519
I20260810 11:42:51.928239 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:51.928948 4035 external_mini_cluster.cc:1428] Setting key 9b47200a365824cead51449335749b16
I20260810 11:42:51.938772 1626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd: Bootstrap starting.
I20260810 11:42:51.942101 1626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:51.943192 1626 log.cc:824] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:42:51 dist-test-slave-drl0 krb5kdc[1501](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362171, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 11:42:51.946326 1626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd: No bootstrap required, opened a new log
I20260810 11:42:51.954556 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:51.931298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46847 (local address 127.0.0.1:45135)
0810 11:42:51.932450 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:51.932469 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:51.932521 (+ 52us) server_negotiation.cc:407] Connection header received
0810 11:42:51.934374 (+ 1853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:51.934397 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:51.934790 (+ 393us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:51.935213 (+ 423us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:51.936728 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:51.938045 (+ 1317us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:51.939415 (+ 1370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:51.940066 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:51.940310 (+ 244us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:51.945461 (+ 5151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:51.945480 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:51.945495 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:51.945532 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:51.949019 (+ 3487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:51.950561 (+ 1542us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:51.950567 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:51.950574 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:51.950771 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:51.951509 (+ 738us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:51.951513 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:51.951515 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:51.952095 (+ 580us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:51.952374 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:51.952970 (+ 596us) server_negotiation.cc:299] Negotiation successful
0810 11:42:51.953322 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":731,"thread_start_us":523,"threads_started":1}
I20260810 11:42:51.956447 1626 raft_consensus.cc:348] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260810 11:42:51.956830 1626 raft_consensus.cc:374] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:51.956962 1626 raft_consensus.cc:729] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d562d329b4164ac1a11cffd8e78456fd, State: Initialized, Role: FOLLOWER
I20260810 11:42:51.957669 1626 consensus_queue.cc:260] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260810 11:42:51.957979 1626 raft_consensus.cc:388] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:51.958142 1626 raft_consensus.cc:482] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:51.958360 1626 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:51.959174 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:42:51.961187 1626 raft_consensus.cc:504] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260810 11:42:51.961750 1626 leader_election.cc:302] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d562d329b4164ac1a11cffd8e78456fd; no voters:
I20260810 11:42:51.962656 1626 leader_election.cc:288] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:51.962920 1634 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:51.965199 1634 raft_consensus.cc:686] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [term 1 LEADER]: Becoming Leader. State: Replica: d562d329b4164ac1a11cffd8e78456fd, State: Running, Role: LEADER
I20260810 11:42:51.966169 1626 sys_catalog.cc:564] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:51.966202 1634 consensus_queue.cc:237] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } }
I20260810 11:42:51.970706 1639 sys_catalog.cc:455] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [sys.catalog]: SysCatalogTable state changed. Reason: New leader d562d329b4164ac1a11cffd8e78456fd. Latest consensus state: current_term: 1 leader_uuid: "d562d329b4164ac1a11cffd8e78456fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260810 11:42:51.971129 1639 sys_catalog.cc:458] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:51.971609 1636 sys_catalog.cc:455] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d562d329b4164ac1a11cffd8e78456fd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d562d329b4164ac1a11cffd8e78456fd" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45135 } } }
I20260810 11:42:51.971994 1636 sys_catalog.cc:458] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:51.989368 1654 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:51.996238 1629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:51.967962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36189 (local address 127.0.0.1:45135)
0810 11:42:51.968225 (+ 263us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:51.968236 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:51.971166 (+ 2930us) server_negotiation.cc:407] Connection header received
0810 11:42:51.971390 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:51.971400 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:51.971517 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:51.971659 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:51.974406 (+ 2747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:51.976008 (+ 1602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:51.978570 (+ 2562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:51.979504 (+ 934us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:51.979628 (+ 124us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:51.985162 (+ 5534us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:51.985174 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:51.985183 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:51.985244 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:51.991842 (+ 6598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:51.992917 (+ 1075us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:51.992924 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:51.992927 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:51.993070 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:51.993701 (+ 631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:51.993707 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:51.993714 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:51.994553 (+ 839us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:51.995035 (+ 482us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:51.995481 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 11:42:51.995678 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:42:51.997126 1654 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:52.010974 1654 catalog_manager.cc:1347] Generated new cluster ID: 96828bb1439b4746a77f069b39a70740
I20260810 11:42:52.011159 1654 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:52.043548 1654 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:52.046108 1654 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:52.059861 1654 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d562d329b4164ac1a11cffd8e78456fd: Generated new TSK 0
I20260810 11:42:52.061111 1654 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:52.077298 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1519
I20260810 11:42:52.081547 1618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:52.182297 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 1519
W20260810 11:42:52.210788 1498 connection.cc:535] client connection to 127.0.0.1:45135 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45135): Connection reset by peer (error 104)
2026-08-10T11:42:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2541 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:52 dist-test-slave-drl0 krb5kdc[1677](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:52 dist-test-slave-drl0 krb5kdc[1677](info): set up 2 sockets
Aug 10 11:42:52 dist-test-slave-drl0 krb5kdc[1677](info): commencing operation
krb5kdc: starting...
W20260810 11:42:53.564579 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:53 dist-test-slave-drl0 krb5kdc[1677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362173, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:42:53.954737 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44113
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44113
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:54.345196 1697 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:54.345880 1697 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:54.346145 1697 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:54.355544 1697 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:54.355720 1697 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:54.355835 1697 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:54.356127 1697 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:54.356237 1697 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:54.367923 1697 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44113
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44113
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:54.369386 1697 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:54.370922 1697 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:54.385156 1702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:54.385213 1703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:54.385391 1706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:54.386202 1697 server_base.cc:1034] running on GCE node
I20260810 11:42:54.387173 1697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:54.388689 1697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:54.389933 1697 hybrid_clock.cc:648] HybridClock initialized: now 1786362174389902 us; error 54 us; skew 500 ppm
I20260810 11:42:54.390419 1697 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:54 dist-test-slave-drl0 krb5kdc[1677](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362174, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:54.394042 1697 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:42:54.396286 1697 webserver.cc:460] Webserver started at http://127.0.0.1:46601/ using document root <none> and password file <none>
I20260810 11:42:54.397204 1697 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:54.397373 1697 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:54.397768 1697 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:54.401494 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "11708a9904c64c4c8235b8ac00283542"
format_stamp: "Formatted at 2026-08-10 11:42:54 on dist-test-slave-drl0"
server_key: "a6738135bfb6e0bf44ece36b7db48d38"
server_key_iv: "97d03e0434898d77a8a6540539c426e0"
server_key_version: "encryptionkey@0"
I20260810 11:42:54.402796 1697 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "11708a9904c64c4c8235b8ac00283542"
format_stamp: "Formatted at 2026-08-10 11:42:54 on dist-test-slave-drl0"
server_key: "a6738135bfb6e0bf44ece36b7db48d38"
server_key_iv: "97d03e0434898d77a8a6540539c426e0"
server_key_version: "encryptionkey@0"
I20260810 11:42:54.409240 1697 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:42:54.413003 1717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:54.414444 1697 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:42:54.414732 1697 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "11708a9904c64c4c8235b8ac00283542"
format_stamp: "Formatted at 2026-08-10 11:42:54 on dist-test-slave-drl0"
server_key: "a6738135bfb6e0bf44ece36b7db48d38"
server_key_iv: "97d03e0434898d77a8a6540539c426e0"
server_key_version: "encryptionkey@0"
I20260810 11:42:54.414980 1697 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:54.429461 1697 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:54.434273 1697 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:54.434574 1697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:54.456737 1697 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44113
I20260810 11:42:54.456895 1791 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44113 every 8 connection(s)
I20260810 11:42:54.458271 1697 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:54.462397 1792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:54.467410 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 1697
I20260810 11:42:54.467602 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:54.468056 4035 external_mini_cluster.cc:1428] Setting key 8c59ab1f959cca956ec6c941579ea712
I20260810 11:42:54.474102 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542: Bootstrap starting.
I20260810 11:42:54.477974 1792 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:54.479446 1792 log.cc:824] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:54.482824 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:54.470084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52581 (local address 127.0.0.1:44113)
0810 11:42:54.470872 (+ 788us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:54.470883 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:54.470977 (+ 94us) server_negotiation.cc:407] Connection header received
0810 11:42:54.472749 (+ 1772us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:54.472771 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:54.473156 (+ 385us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:54.473461 (+ 305us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:54.474129 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:54.475546 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:54.477503 (+ 1957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:54.478294 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:54.478456 (+ 162us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:54.479922 (+ 1466us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:54.479930 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:54.479944 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:54.480903 (+ 959us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:54.481026 (+ 123us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:54.481063 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:54.481232 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:54.481647 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 11:42:54.481849 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":569,"thread_start_us":352,"threads_started":1}
I20260810 11:42:54.483091 1792 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542: No bootstrap required, opened a new log
I20260810 11:42:54.488648 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:42:54.492730 1792 raft_consensus.cc:348] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260810 11:42:54.493284 1792 raft_consensus.cc:374] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:54.493395 1792 raft_consensus.cc:729] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 11708a9904c64c4c8235b8ac00283542, State: Initialized, Role: FOLLOWER
I20260810 11:42:54.494168 1792 consensus_queue.cc:260] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260810 11:42:54.494496 1792 raft_consensus.cc:388] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:54.494652 1792 raft_consensus.cc:482] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:54.494865 1792 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:54.499260 1792 raft_consensus.cc:504] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260810 11:42:54.499804 1792 leader_election.cc:302] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 11708a9904c64c4c8235b8ac00283542; no voters:
I20260810 11:42:54.500571 1792 leader_election.cc:288] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:54.500880 1806 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:54.502499 1806 raft_consensus.cc:686] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [term 1 LEADER]: Becoming Leader. State: Replica: 11708a9904c64c4c8235b8ac00283542, State: Running, Role: LEADER
I20260810 11:42:54.503328 1792 sys_catalog.cc:564] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:54.503288 1806 consensus_queue.cc:237] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } }
I20260810 11:42:54.504215 1795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:54.494434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56115 (local address 127.0.0.1:44113)
0810 11:42:54.494636 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:54.494648 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:54.495270 (+ 622us) server_negotiation.cc:407] Connection header received
0810 11:42:54.495468 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:54.495473 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:54.495556 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:54.496107 (+ 551us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:54.496715 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:54.498138 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:54.500107 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:54.501101 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:54.501252 (+ 151us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:54.503070 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:54.503076 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:54.503081 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:54.503103 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:54.503290 (+ 187us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:54.503325 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:54.503499 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:54.503804 (+ 305us) server_negotiation.cc:299] Negotiation successful
0810 11:42:54.503862 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260810 11:42:54.507143 1809 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 11708a9904c64c4c8235b8ac00283542. Latest consensus state: current_term: 1 leader_uuid: "11708a9904c64c4c8235b8ac00283542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } } }
I20260810 11:42:54.507123 1808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "11708a9904c64c4c8235b8ac00283542" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "11708a9904c64c4c8235b8ac00283542" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44113 } } }
I20260810 11:42:54.507682 1809 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:54.507682 1808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:54.509877 1815 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:54.515710 1815 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:54.530061 1815 catalog_manager.cc:1347] Generated new cluster ID: dd60eb01dd214a55a0861f374a3a2a46
I20260810 11:42:54.530179 1815 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:54.540673 1815 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:54.542907 1815 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:54.551540 1815 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 11708a9904c64c4c8235b8ac00283542: Generated new TSK 0
I20260810 11:42:54.552472 1815 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:54.561332 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1697
W20260810 11:42:54.562816 1787 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:52581
W20260810 11:42:54.564181 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1697: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:42:54.564400 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 1697
W20260810 11:42:54.597016 1673 connection.cc:535] client connection to 127.0.0.1:44113 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44113): Connection reset by peer (error 104)
2026-08-10T11:42:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2384 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:54 dist-test-slave-drl0 krb5kdc[1838](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:54 dist-test-slave-drl0 krb5kdc[1838](info): set up 2 sockets
Aug 10 11:42:54 dist-test-slave-drl0 krb5kdc[1838](info): commencing operation
krb5kdc: starting...
W20260810 11:42:55.964046 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.004s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:56 dist-test-slave-drl0 krb5kdc[1838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362176, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:42:56.347731 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42825
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42825
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:56.720896 1856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:56.721261 1856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:56.721437 1856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:56.728998 1856 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:56.729158 1856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:56.729277 1856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:56.729391 1856 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:56.729477 1856 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:56.740618 1856 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42825
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:56.742049 1856 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:56.743424 1856 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:56.756583 1869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:56.756984 1865 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:56.757014 1863 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:56.757970 1856 server_base.cc:1034] running on GCE node
I20260810 11:42:56.758773 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:56.760299 1856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:56.761559 1856 hybrid_clock.cc:648] HybridClock initialized: now 1786362176761513 us; error 40 us; skew 500 ppm
I20260810 11:42:56.761951 1856 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:56 dist-test-slave-drl0 krb5kdc[1838](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362176, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:56.765514 1856 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:42:56.767853 1856 webserver.cc:460] Webserver started at http://127.0.0.1:44439/ using document root <none> and password file <none>
I20260810 11:42:56.768817 1856 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:56.768993 1856 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:56.769408 1856 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:56.773247 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "c99f5140050e4130a667218df31edd6c"
format_stamp: "Formatted at 2026-08-10 11:42:56 on dist-test-slave-drl0"
server_key: "323ab44733b794e71e1037328143adf2"
server_key_iv: "d3a3978c5f9697ab62e8dd866bc4fd84"
server_key_version: "encryptionkey@0"
I20260810 11:42:56.774636 1856 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "c99f5140050e4130a667218df31edd6c"
format_stamp: "Formatted at 2026-08-10 11:42:56 on dist-test-slave-drl0"
server_key: "323ab44733b794e71e1037328143adf2"
server_key_iv: "d3a3978c5f9697ab62e8dd866bc4fd84"
server_key_version: "encryptionkey@0"
I20260810 11:42:56.781188 1856 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260810 11:42:56.785624 1876 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:56.787667 1856 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260810 11:42:56.787927 1856 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "c99f5140050e4130a667218df31edd6c"
format_stamp: "Formatted at 2026-08-10 11:42:56 on dist-test-slave-drl0"
server_key: "323ab44733b794e71e1037328143adf2"
server_key_iv: "d3a3978c5f9697ab62e8dd866bc4fd84"
server_key_version: "encryptionkey@0"
I20260810 11:42:56.788429 1856 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:56.795770 1856 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:56.800817 1856 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:56.801154 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:56.825693 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42825
I20260810 11:42:56.825714 1953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42825 every 8 connection(s)
I20260810 11:42:56.827260 1856 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:56.831106 1955 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:56.835990 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 1856
I20260810 11:42:56.836195 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:56.836795 4035 external_mini_cluster.cc:1428] Setting key 18109e6d199dbecd343a1d18ab6987d8
I20260810 11:42:56.843438 1955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c: Bootstrap starting.
I20260810 11:42:56.845542 1960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:56.838788 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32919 (local address 127.0.0.1:42825)
0810 11:42:56.839636 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:56.839651 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:56.839699 (+ 48us) server_negotiation.cc:407] Connection header received
0810 11:42:56.841756 (+ 2057us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:56.841779 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:56.842315 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:56.842784 (+ 469us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:56.843976 (+ 1192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:56.843989 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:56.844001 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:56.844016 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:56.844077 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:56.844097 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:56.844322 (+ 225us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:56.844468 (+ 146us) server_negotiation.cc:299] Negotiation successful
0810 11:42:56.844650 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":360,"threads_started":1}
I20260810 11:42:56.846760 1955 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:56.848191 1955 log.cc:824] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:56.850098 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:42:56.852416 1955 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c: No bootstrap required, opened a new log
I20260810 11:42:56.860684 1960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:56.854851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49303 (local address 127.0.0.1:42825)
0810 11:42:56.855093 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:56.855099 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:56.855408 (+ 309us) server_negotiation.cc:407] Connection header received
0810 11:42:56.855618 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:56.855623 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:56.855678 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:56.855785 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:56.859561 (+ 3776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:56.859565 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:56.859569 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:42:56.859585 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:56.859631 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:42:56.859647 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:56.859756 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:56.859899 (+ 143us) server_negotiation.cc:299] Negotiation successful
0810 11:42:56.859951 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
I20260810 11:42:56.862039 1955 raft_consensus.cc:348] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } }
I20260810 11:42:56.862601 1955 raft_consensus.cc:374] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:56.862720 1955 raft_consensus.cc:729] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c99f5140050e4130a667218df31edd6c, State: Initialized, Role: FOLLOWER
I20260810 11:42:56.863333 1955 consensus_queue.cc:260] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } }
I20260810 11:42:56.863631 1955 raft_consensus.cc:388] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:56.863755 1955 raft_consensus.cc:482] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:56.863941 1955 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:56.866645 1955 raft_consensus.cc:504] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } }
I20260810 11:42:56.867188 1955 leader_election.cc:302] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c99f5140050e4130a667218df31edd6c; no voters:
I20260810 11:42:56.867821 1955 leader_election.cc:288] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:56.868096 1967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:56.869078 1967 raft_consensus.cc:686] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [term 1 LEADER]: Becoming Leader. State: Replica: c99f5140050e4130a667218df31edd6c, State: Running, Role: LEADER
I20260810 11:42:56.869743 1967 consensus_queue.cc:237] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } }
I20260810 11:42:56.870093 1955 sys_catalog.cc:564] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:56.872668 1973 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [sys.catalog]: SysCatalogTable state changed. Reason: New leader c99f5140050e4130a667218df31edd6c. Latest consensus state: current_term: 1 leader_uuid: "c99f5140050e4130a667218df31edd6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } } }
I20260810 11:42:56.873286 1973 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:56.872640 1972 sys_catalog.cc:455] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c99f5140050e4130a667218df31edd6c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c99f5140050e4130a667218df31edd6c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42825 } } }
I20260810 11:42:56.874100 1972 sys_catalog.cc:458] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:56.874392 1981 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:56.880364 1981 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:56.891250 1981 catalog_manager.cc:1347] Generated new cluster ID: 914db67b691c462098828382991c3091
I20260810 11:42:56.891377 1981 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:56.944419 1981 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:56.945006 1981 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:56.958608 1981 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c99f5140050e4130a667218df31edd6c: Generated new TSK 0
I20260810 11:42:56.959627 1981 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:56.971400 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1856
W20260810 11:42:56.972652 1948 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:32919
W20260810 11:42:56.973433 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 1856: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:42:56.973603 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 1856
2026-08-10T11:42:57Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2410 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:57 dist-test-slave-drl0 krb5kdc[2005](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:57 dist-test-slave-drl0 krb5kdc[2005](info): set up 2 sockets
Aug 10 11:42:57 dist-test-slave-drl0 krb5kdc[2005](info): commencing operation
krb5kdc: starting...
W20260810 11:42:58.369793 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.343s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:42:58 dist-test-slave-drl0 krb5kdc[2005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:42:58Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:42:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:42:58.754778 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44735
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:36575
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44735
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:42:59.130843 2022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:42:59.131264 2022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:42:59.131438 2022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:42:59.139237 2022 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:42:59.139484 2022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:42:59.139623 2022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:42:59.139768 2022 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:42:59.139854 2022 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:42:59.150601 2022 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:36575
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44735
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44735
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:42:59.152014 2022 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:42:59.153535 2022 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:42:59.168833 2034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:59.168803 2032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:42:59.169102 2031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:42:59.169957 2022 server_base.cc:1034] running on GCE node
I20260810 11:42:59.171236 2022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:42:59.173079 2022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:42:59.174358 2022 hybrid_clock.cc:648] HybridClock initialized: now 1786362179174245 us; error 86 us; skew 500 ppm
I20260810 11:42:59.174811 2022 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:42:59 dist-test-slave-drl0 krb5kdc[2005](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:42:59.178448 2022 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:42:59.180594 2022 webserver.cc:460] Webserver started at http://127.0.0.1:46487/ using document root <none> and password file <none>
I20260810 11:42:59.181360 2022 fs_manager.cc:359] Metadata directory not provided
I20260810 11:42:59.181524 2022 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:42:59.181854 2022 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:42:59.185438 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "4d60e5038b7e44999fa3543ce2ab3a68"
format_stamp: "Formatted at 2026-08-10 11:42:59 on dist-test-slave-drl0"
server_key: "56aacc300e20ad5acb0bfafeccd30345"
server_key_iv: "efe41d29bfa244e400ef79cba46e6576"
server_key_version: "encryptionkey@0"
I20260810 11:42:59.186255 2022 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "4d60e5038b7e44999fa3543ce2ab3a68"
format_stamp: "Formatted at 2026-08-10 11:42:59 on dist-test-slave-drl0"
server_key: "56aacc300e20ad5acb0bfafeccd30345"
server_key_iv: "efe41d29bfa244e400ef79cba46e6576"
server_key_version: "encryptionkey@0"
I20260810 11:42:59.193220 2022 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.004s
I20260810 11:42:59.197326 2043 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:42:59.198798 2022 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:42:59.199120 2022 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "4d60e5038b7e44999fa3543ce2ab3a68"
format_stamp: "Formatted at 2026-08-10 11:42:59 on dist-test-slave-drl0"
server_key: "56aacc300e20ad5acb0bfafeccd30345"
server_key_iv: "efe41d29bfa244e400ef79cba46e6576"
server_key_version: "encryptionkey@0"
I20260810 11:42:59.199352 2022 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:42:59.234227 2022 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:42:59.239061 2022 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:42:59.239385 2022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:42:59.260802 2022 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44735
I20260810 11:42:59.260834 2113 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44735 every 8 connection(s)
I20260810 11:42:59.262393 2022 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:42:59.266992 2115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:42:59.271742 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2022
I20260810 11:42:59.271960 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:42:59.272428 4035 external_mini_cluster.cc:1428] Setting key 7c80e61a240a8770e121d0d4e6f9296f
I20260810 11:42:59.279393 2115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68: Bootstrap starting.
I20260810 11:42:59.282753 2115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68: Neither blocks nor log segments found. Creating new log.
I20260810 11:42:59.283776 2115 log.cc:824] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68: Log is configured to *not* fsync() on all Append() calls
I20260810 11:42:59.286680 2115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68: No bootstrap required, opened a new log
Aug 10 11:42:59 dist-test-slave-drl0 krb5kdc[2005](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362178, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 11:42:59.295154 2115 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } }
I20260810 11:42:59.295544 2115 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:42:59.295670 2115 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d60e5038b7e44999fa3543ce2ab3a68, State: Initialized, Role: FOLLOWER
I20260810 11:42:59.297262 2115 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } }
I20260810 11:42:59.297561 2115 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:42:59.297760 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:59.275064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56191 (local address 127.0.0.1:44735)
0810 11:42:59.276008 (+ 944us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:59.276029 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:59.276096 (+ 67us) server_negotiation.cc:407] Connection header received
0810 11:42:59.278515 (+ 2419us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:59.278542 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:59.279063 (+ 521us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:59.279514 (+ 451us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:59.280958 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:59.282299 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:59.283721 (+ 1422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:59.284465 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:59.284685 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:59.289405 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:59.289413 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:59.289430 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:59.289465 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:59.293222 (+ 3757us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:59.294020 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:59.294030 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:59.294045 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:59.294227 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:59.294651 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:59.294657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:59.294660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:59.295292 (+ 632us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:59.295617 (+ 325us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:59.296142 (+ 525us) server_negotiation.cc:299] Negotiation successful
0810 11:42:59.296612 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":689,"thread_start_us":407,"threads_started":1}
I20260810 11:42:59.297739 2115 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:42:59.298005 2115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:42:59.301044 2115 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } }
I20260810 11:42:59.301576 2115 leader_election.cc:302] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d60e5038b7e44999fa3543ce2ab3a68; no voters:
I20260810 11:42:59.302174 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:42:59.302323 2115 leader_election.cc:288] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:42:59.302598 2121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:42:59.303936 2121 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [term 1 LEADER]: Becoming Leader. State: Replica: 4d60e5038b7e44999fa3543ce2ab3a68, State: Running, Role: LEADER
I20260810 11:42:59.304639 2115 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:42:59.304579 2121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } }
I20260810 11:42:59.310911 2123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } } }
I20260810 11:42:59.311424 2123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:59.312104 2124 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d60e5038b7e44999fa3543ce2ab3a68. Latest consensus state: current_term: 1 leader_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d60e5038b7e44999fa3543ce2ab3a68" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44735 } } }
I20260810 11:42:59.312462 2124 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68 [sys.catalog]: This master's current role is: LEADER
I20260810 11:42:59.312438 2144 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:42:59.324316 2144 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:42:59.334959 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:42:59.310482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50115 (local address 127.0.0.1:44735)
0810 11:42:59.310709 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0810 11:42:59.310718 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:42:59.313255 (+ 2537us) server_negotiation.cc:407] Connection header received
0810 11:42:59.313627 (+ 372us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:42:59.313635 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:42:59.313721 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:42:59.313799 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:42:59.316287 (+ 2488us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:59.317760 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:59.320272 (+ 2512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:42:59.321303 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:42:59.321486 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:42:59.325447 (+ 3961us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:42:59.325461 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:42:59.325472 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:42:59.325526 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:42:59.330656 (+ 5130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:59.331822 (+ 1166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:59.331831 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:59.331838 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:59.332225 (+ 387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:42:59.333020 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:42:59.333032 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:42:59.333036 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:42:59.333462 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:42:59.333692 (+ 230us) server_negotiation.cc:1035] Waiting for connection context
0810 11:42:59.334303 (+ 611us) server_negotiation.cc:299] Negotiation successful
0810 11:42:59.334546 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260810 11:42:59.342664 2144 catalog_manager.cc:1347] Generated new cluster ID: 26aaf0372c7a4597bdac05e8c9265050
I20260810 11:42:59.342818 2144 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:42:59.351455 2144 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:42:59.353235 2144 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:42:59.360965 2144 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4d60e5038b7e44999fa3543ce2ab3a68: Generated new TSK 0
I20260810 11:42:59.362003 2144 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:42:59.371783 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2022
I20260810 11:42:59.372956 2107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:42:59.472239 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2022
2026-08-10T11:42:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2495 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:42:59 dist-test-slave-drl0 krb5kdc[2169](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:42:59 dist-test-slave-drl0 krb5kdc[2169](info): set up 2 sockets
Aug 10 11:42:59 dist-test-slave-drl0 krb5kdc[2169](info): commencing operation
krb5kdc: starting...
W20260810 11:43:00.865630 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.344s user 0.000s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:01 dist-test-slave-drl0 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362181, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:01.242259 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44833
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:01.632205 2187 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:01.632586 2187 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:01.632764 2187 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:01.640528 2187 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:01.640703 2187 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:01.640815 2187 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:01.640934 2187 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:01.641041 2187 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:01.651440 2187 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44833
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44833
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:01.652882 2187 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:01.654263 2187 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:01.668687 2192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:01.668653 2194 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:01.668605 2197 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:01.671077 2187 server_base.cc:1034] running on GCE node
I20260810 11:43:01.671916 2187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:01.673465 2187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:01.674861 2187 hybrid_clock.cc:648] HybridClock initialized: now 1786362181674815 us; error 32 us; skew 500 ppm
I20260810 11:43:01.675264 2187 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:01 dist-test-slave-drl0 krb5kdc[2169](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:01.678486 2187 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:01.679978 2187 webserver.cc:460] Webserver started at http://127.0.0.1:34641/ using document root <none> and password file <none>
I20260810 11:43:01.680649 2187 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:01.680758 2187 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:01.681039 2187 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:01.684259 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "90b95b4222cd4ee2a1dab917d67e4604"
format_stamp: "Formatted at 2026-08-10 11:43:01 on dist-test-slave-drl0"
server_key: "3ff5fb772296503486260bd1b5a477cc"
server_key_iv: "1802fdbf3e73dc948b4c01bb044506bd"
server_key_version: "encryptionkey@0"
I20260810 11:43:01.685007 2187 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "90b95b4222cd4ee2a1dab917d67e4604"
format_stamp: "Formatted at 2026-08-10 11:43:01 on dist-test-slave-drl0"
server_key: "3ff5fb772296503486260bd1b5a477cc"
server_key_iv: "1802fdbf3e73dc948b4c01bb044506bd"
server_key_version: "encryptionkey@0"
I20260810 11:43:01.690233 2187 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260810 11:43:01.693789 2206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:01.695360 2187 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:43:01.695555 2187 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "90b95b4222cd4ee2a1dab917d67e4604"
format_stamp: "Formatted at 2026-08-10 11:43:01 on dist-test-slave-drl0"
server_key: "3ff5fb772296503486260bd1b5a477cc"
server_key_iv: "1802fdbf3e73dc948b4c01bb044506bd"
server_key_version: "encryptionkey@0"
I20260810 11:43:01.695772 2187 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:01.715690 2187 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:01.720405 2187 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:01.720775 2187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:01.742553 2187 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44833
I20260810 11:43:01.742734 2273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44833 every 8 connection(s)
I20260810 11:43:01.744138 2187 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:01.748632 2275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:01.749588 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2187
I20260810 11:43:01.749787 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:01.750250 4035 external_mini_cluster.cc:1428] Setting key 15dfd15d08bc7a1eac0c21fb9f8e5de6
I20260810 11:43:01.765357 2275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604: Bootstrap starting.
I20260810 11:43:01.766987 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:01.752323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54339 (local address 127.0.0.1:44833)
0810 11:43:01.753106 (+ 783us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:01.753121 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:01.753170 (+ 49us) server_negotiation.cc:407] Connection header received
0810 11:43:01.755696 (+ 2526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:01.755725 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:01.756239 (+ 514us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:01.756700 (+ 461us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:01.757216 (+ 516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:01.759122 (+ 1906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:01.760455 (+ 1333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:01.762177 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:01.762738 (+ 561us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:01.763782 (+ 1044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:01.763798 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:01.763821 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:01.764574 (+ 753us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:01.764691 (+ 117us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:01.764737 (+ 46us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:01.764893 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:01.765308 (+ 415us) server_negotiation.cc:299] Negotiation successful
0810 11:43:01.765593 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":328,"threads_started":1}
I20260810 11:43:01.769680 2275 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:01.771021 2275 log.cc:824] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:01.771329 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:01.774488 2275 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604: No bootstrap required, opened a new log
I20260810 11:43:01.786018 2279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:01.777329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49709 (local address 127.0.0.1:44833)
0810 11:43:01.777706 (+ 377us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:01.777716 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:01.777920 (+ 204us) server_negotiation.cc:407] Connection header received
0810 11:43:01.778114 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:01.778121 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:01.778256 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:01.778435 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:01.779103 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:01.780208 (+ 1105us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:01.782443 (+ 2235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:01.783508 (+ 1065us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:01.783665 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:01.785094 (+ 1429us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:01.785104 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:01.785110 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:01.785142 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:01.785193 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:01.785215 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:01.785328 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:01.785560 (+ 232us) server_negotiation.cc:299] Negotiation successful
0810 11:43:01.785669 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:43:01.787771 2275 raft_consensus.cc:348] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260810 11:43:01.788249 2275 raft_consensus.cc:374] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:01.788367 2275 raft_consensus.cc:729] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 90b95b4222cd4ee2a1dab917d67e4604, State: Initialized, Role: FOLLOWER
I20260810 11:43:01.789079 2275 consensus_queue.cc:260] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260810 11:43:01.789328 2275 raft_consensus.cc:388] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:01.789482 2275 raft_consensus.cc:482] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:01.789660 2275 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:01.792146 2275 raft_consensus.cc:504] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260810 11:43:01.792671 2275 leader_election.cc:302] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 90b95b4222cd4ee2a1dab917d67e4604; no voters:
I20260810 11:43:01.793265 2275 leader_election.cc:288] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:01.793408 2293 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:01.794180 2293 raft_consensus.cc:686] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [term 1 LEADER]: Becoming Leader. State: Replica: 90b95b4222cd4ee2a1dab917d67e4604, State: Running, Role: LEADER
I20260810 11:43:01.794734 2293 consensus_queue.cc:237] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } }
I20260810 11:43:01.795535 2275 sys_catalog.cc:564] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:01.797587 2295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 90b95b4222cd4ee2a1dab917d67e4604. Latest consensus state: current_term: 1 leader_uuid: "90b95b4222cd4ee2a1dab917d67e4604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260810 11:43:01.797909 2295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:01.798894 2298 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:01.798825 2294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "90b95b4222cd4ee2a1dab917d67e4604" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "90b95b4222cd4ee2a1dab917d67e4604" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44833 } } }
I20260810 11:43:01.799255 2294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:01.804901 2298 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:01.815387 2298 catalog_manager.cc:1347] Generated new cluster ID: 82c644b0ad9a4870acaa585ceffac657
I20260810 11:43:01.815508 2298 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:01.837879 2298 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:01.839473 2298 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:01.850198 2298 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 90b95b4222cd4ee2a1dab917d67e4604: Generated new TSK 0
I20260810 11:43:01.851114 2298 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:01.872229 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2187
W20260810 11:43:01.873517 2268 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54339
W20260810 11:43:01.874229 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2187: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:01.874392 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2187
2026-08-10T11:43:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2406 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:02 dist-test-slave-drl0 krb5kdc[2329](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:02 dist-test-slave-drl0 krb5kdc[2329](info): set up 2 sockets
Aug 10 11:43:02 dist-test-slave-drl0 krb5kdc[2329](info): commencing operation
krb5kdc: starting...
W20260810 11:43:03.275549 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.346s user 0.009s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:03 dist-test-slave-drl0 krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362183, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:03.646570 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34063
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37215
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34063
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:04.030467 2349 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:04.030778 2349 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:04.030879 2349 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:04.039242 2349 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:04.039443 2349 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:04.039535 2349 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:04.039597 2349 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:04.039642 2349 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:04.052438 2349 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37215
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:04.053984 2349 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:04.055320 2349 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:04.069442 2357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:04.069671 2359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:04.069826 2363 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:04.070669 2349 server_base.cc:1034] running on GCE node
I20260810 11:43:04.071671 2349 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:04.073122 2349 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:04.074452 2349 hybrid_clock.cc:648] HybridClock initialized: now 1786362184074313 us; error 77 us; skew 500 ppm
I20260810 11:43:04.074846 2349 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:04 dist-test-slave-drl0 krb5kdc[2329](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:04.078188 2349 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:04.080152 2349 webserver.cc:460] Webserver started at http://127.0.0.1:45081/ using document root <none> and password file <none>
I20260810 11:43:04.080906 2349 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:04.081065 2349 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:04.081393 2349 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:04.084805 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "d796b6405abe4e3698d24b8d016f9f35"
format_stamp: "Formatted at 2026-08-10 11:43:04 on dist-test-slave-drl0"
server_key: "cb1b9026bb91cbc60a67ef4b43f01e12"
server_key_iv: "854db795ff0ce55cc1b6fae839c125b9"
server_key_version: "encryptionkey@0"
I20260810 11:43:04.085606 2349 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "d796b6405abe4e3698d24b8d016f9f35"
format_stamp: "Formatted at 2026-08-10 11:43:04 on dist-test-slave-drl0"
server_key: "cb1b9026bb91cbc60a67ef4b43f01e12"
server_key_iv: "854db795ff0ce55cc1b6fae839c125b9"
server_key_version: "encryptionkey@0"
I20260810 11:43:04.091178 2349 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260810 11:43:04.094888 2372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:04.096441 2349 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:43:04.096649 2349 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "d796b6405abe4e3698d24b8d016f9f35"
format_stamp: "Formatted at 2026-08-10 11:43:04 on dist-test-slave-drl0"
server_key: "cb1b9026bb91cbc60a67ef4b43f01e12"
server_key_iv: "854db795ff0ce55cc1b6fae839c125b9"
server_key_version: "encryptionkey@0"
I20260810 11:43:04.096858 2349 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:04.103374 2349 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:04.108055 2349 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:04.108422 2349 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:04.131286 2349 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34063
I20260810 11:43:04.131367 2447 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34063 every 8 connection(s)
I20260810 11:43:04.132716 2349 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:04.135042 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2349
I20260810 11:43:04.135267 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:04.135833 4035 external_mini_cluster.cc:1428] Setting key e131ba0c91bbe1ec204dc56169da3438
I20260810 11:43:04.137269 2449 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:04.144181 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:04.138004 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51681 (local address 127.0.0.1:34063)
0810 11:43:04.138873 (+ 869us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:04.138884 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:04.138979 (+ 95us) server_negotiation.cc:407] Connection header received
0810 11:43:04.140636 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:04.140656 (+ 20us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:04.141022 (+ 366us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:04.141365 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:04.142521 (+ 1156us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:04.142528 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:04.142540 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:04.142565 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:04.142643 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:04.142667 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:04.142879 (+ 212us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:04.143076 (+ 197us) server_negotiation.cc:299] Negotiation successful
0810 11:43:04.143305 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":405,"threads_started":1}
I20260810 11:43:04.148594 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:04.151980 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35: Bootstrap starting.
I20260810 11:43:04.156038 2449 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:04.156229 2453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:04.153157 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39293 (local address 127.0.0.1:34063)
0810 11:43:04.153365 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:04.153371 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:04.153762 (+ 391us) server_negotiation.cc:407] Connection header received
0810 11:43:04.153940 (+ 178us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:04.153945 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:04.153998 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:04.154096 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:04.155270 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:04.155274 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:04.155277 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:04.155292 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:04.155338 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:04.155354 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:04.155575 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:04.155746 (+ 171us) server_negotiation.cc:299] Negotiation successful
0810 11:43:04.155796 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:43:04.157275 2449 log.cc:824] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:04.160395 2449 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35: No bootstrap required, opened a new log
I20260810 11:43:04.169310 2449 raft_consensus.cc:348] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } }
I20260810 11:43:04.169768 2449 raft_consensus.cc:374] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:04.169916 2449 raft_consensus.cc:729] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d796b6405abe4e3698d24b8d016f9f35, State: Initialized, Role: FOLLOWER
I20260810 11:43:04.170674 2449 consensus_queue.cc:260] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } }
I20260810 11:43:04.170987 2449 raft_consensus.cc:388] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:04.171139 2449 raft_consensus.cc:482] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:04.171296 2449 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:04.173566 2449 raft_consensus.cc:504] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } }
I20260810 11:43:04.174039 2449 leader_election.cc:302] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d796b6405abe4e3698d24b8d016f9f35; no voters:
I20260810 11:43:04.174820 2449 leader_election.cc:288] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:04.175215 2461 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:04.176410 2461 raft_consensus.cc:686] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [term 1 LEADER]: Becoming Leader. State: Replica: d796b6405abe4e3698d24b8d016f9f35, State: Running, Role: LEADER
I20260810 11:43:04.177062 2461 consensus_queue.cc:237] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } }
I20260810 11:43:04.177268 2449 sys_catalog.cc:564] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:04.180651 2463 sys_catalog.cc:455] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d796b6405abe4e3698d24b8d016f9f35. Latest consensus state: current_term: 1 leader_uuid: "d796b6405abe4e3698d24b8d016f9f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } } }
I20260810 11:43:04.181106 2463 sys_catalog.cc:458] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:04.180861 2462 sys_catalog.cc:455] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d796b6405abe4e3698d24b8d016f9f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d796b6405abe4e3698d24b8d016f9f35" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34063 } } }
I20260810 11:43:04.181280 2462 sys_catalog.cc:458] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:04.184038 2472 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:04.190289 2472 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:04.200112 2472 catalog_manager.cc:1347] Generated new cluster ID: 191c5efd77df4a1ab808d2e1afcab381
I20260810 11:43:04.200246 2472 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:04.219885 2472 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:04.220531 2472 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:04.246302 2472 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d796b6405abe4e3698d24b8d016f9f35: Generated new TSK 0
I20260810 11:43:04.247619 2472 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:04.258739 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2349
W20260810 11:43:04.260139 2441 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:51681
W20260810 11:43:04.261097 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2349: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:04.261350 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2349
2026-08-10T11:43:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2381 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:04 dist-test-slave-drl0 krb5kdc[2489](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:04 dist-test-slave-drl0 krb5kdc[2489](info): set up 2 sockets
Aug 10 11:43:04 dist-test-slave-drl0 krb5kdc[2489](info): commencing operation
krb5kdc: starting...
W20260810 11:43:05.646788 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.335s user 0.003s sys 0.047s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:05 dist-test-slave-drl0 krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:06.010802 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46643
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33721
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46643
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:06.380561 2508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:06.380841 2508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:06.381021 2508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:06.388731 2508 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:06.389014 2508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:06.389117 2508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:06.389233 2508 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:06.389323 2508 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:06.401686 2508 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33721
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:06.403187 2508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:06.404577 2508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:06.417819 2513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:06.417619 2514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:06.418012 2517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:06.419270 2508 server_base.cc:1034] running on GCE node
I20260810 11:43:06.420083 2508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:06.421744 2508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:06.423048 2508 hybrid_clock.cc:648] HybridClock initialized: now 1786362186423006 us; error 28 us; skew 500 ppm
I20260810 11:43:06.423430 2508 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:06 dist-test-slave-drl0 krb5kdc[2489](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362186, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:06.428066 2508 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:06.429878 2508 webserver.cc:460] Webserver started at http://127.0.0.1:38977/ using document root <none> and password file <none>
I20260810 11:43:06.430601 2508 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:06.430773 2508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:06.431164 2508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:06.434211 2508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "ccccb34fe1994d5588961cea0559bf97"
format_stamp: "Formatted at 2026-08-10 11:43:06 on dist-test-slave-drl0"
server_key: "74db5feac6ce8f8fb9ba40f15e953a33"
server_key_iv: "35f45ffb9e55c8e7da5544001d3c5e72"
server_key_version: "encryptionkey@0"
I20260810 11:43:06.435048 2508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "ccccb34fe1994d5588961cea0559bf97"
format_stamp: "Formatted at 2026-08-10 11:43:06 on dist-test-slave-drl0"
server_key: "74db5feac6ce8f8fb9ba40f15e953a33"
server_key_iv: "35f45ffb9e55c8e7da5544001d3c5e72"
server_key_version: "encryptionkey@0"
I20260810 11:43:06.442340 2508 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260810 11:43:06.446642 2526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:06.448153 2508 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260810 11:43:06.448451 2508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "ccccb34fe1994d5588961cea0559bf97"
format_stamp: "Formatted at 2026-08-10 11:43:06 on dist-test-slave-drl0"
server_key: "74db5feac6ce8f8fb9ba40f15e953a33"
server_key_iv: "35f45ffb9e55c8e7da5544001d3c5e72"
server_key_version: "encryptionkey@0"
I20260810 11:43:06.448671 2508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:06.467468 2508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:06.471923 2508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:06.472247 2508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:06.493885 2508 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46643
I20260810 11:43:06.493907 2610 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46643 every 8 connection(s)
I20260810 11:43:06.495371 2508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:06.497038 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2508
I20260810 11:43:06.497236 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:06.497848 4035 external_mini_cluster.cc:1428] Setting key 5ef175c0ece4a5a593906adb74bf1019
I20260810 11:43:06.499523 2611 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:06.512977 2611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97: Bootstrap starting.
Aug 10 11:43:06 dist-test-slave-drl0 krb5kdc[2489](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362185, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 11:43:06.517207 2611 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:06.518296 2611 log.cc:824] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:06.522159 2611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97: No bootstrap required, opened a new log
I20260810 11:43:06.527791 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:06.500845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:58697 (local address 127.0.0.1:46643)
0810 11:43:06.501743 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:06.501755 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:06.501801 (+ 46us) server_negotiation.cc:407] Connection header received
0810 11:43:06.505004 (+ 3203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:06.505065 (+ 61us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:06.505721 (+ 656us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:06.506345 (+ 624us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:06.507910 (+ 1565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:06.509202 (+ 1292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:06.510978 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:06.511899 (+ 921us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:06.512204 (+ 305us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:06.517829 (+ 5625us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:06.517839 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:06.517856 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:06.517905 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:06.522237 (+ 4332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:06.523177 (+ 940us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:06.523186 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:06.523194 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:06.523415 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:06.523965 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:06.523975 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:06.523979 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:06.524751 (+ 772us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:06.525114 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:06.525878 (+ 764us) server_negotiation.cc:299] Negotiation successful
0810 11:43:06.526245 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":385,"threads_started":1}
I20260810 11:43:06.532343 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:06.532606 2611 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260810 11:43:06.532974 2611 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:06.533078 2611 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccccb34fe1994d5588961cea0559bf97, State: Initialized, Role: FOLLOWER
I20260810 11:43:06.533685 2611 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260810 11:43:06.533922 2611 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:06.534070 2611 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:06.534245 2611 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:06.536849 2611 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260810 11:43:06.537503 2611 leader_election.cc:302] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccccb34fe1994d5588961cea0559bf97; no voters:
I20260810 11:43:06.538494 2625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:06.538688 2611 leader_election.cc:288] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:06.539364 2625 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [term 1 LEADER]: Becoming Leader. State: Replica: ccccb34fe1994d5588961cea0559bf97, State: Running, Role: LEADER
I20260810 11:43:06.540071 2625 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } }
I20260810 11:43:06.542217 2628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccccb34fe1994d5588961cea0559bf97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } } }
I20260810 11:43:06.542726 2628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:06.543186 2611 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:06.543087 2627 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccccb34fe1994d5588961cea0559bf97. Latest consensus state: current_term: 1 leader_uuid: "ccccb34fe1994d5588961cea0559bf97" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccccb34fe1994d5588961cea0559bf97" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46643 } } }
I20260810 11:43:06.543467 2627 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:06.556454 2631 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:06.560688 2616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:06.537356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:52513 (local address 127.0.0.1:46643)
0810 11:43:06.537602 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:06.537611 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:06.538050 (+ 439us) server_negotiation.cc:407] Connection header received
0810 11:43:06.538576 (+ 526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:06.538585 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:06.538682 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:06.538812 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:06.541074 (+ 2262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:06.542336 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:06.544292 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:06.545457 (+ 1165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:06.545635 (+ 178us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:06.550505 (+ 4870us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:06.550517 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:06.550525 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:06.550580 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:06.555727 (+ 5147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:06.557593 (+ 1866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:06.557600 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:06.557604 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:06.557760 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:06.558447 (+ 687us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:06.558456 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:06.558461 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:06.558957 (+ 496us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:06.559255 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:06.559870 (+ 615us) server_negotiation.cc:299] Negotiation successful
0810 11:43:06.560141 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 11:43:06.563364 2631 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:06.574438 2631 catalog_manager.cc:1347] Generated new cluster ID: bd61aa7929ba45d48c2dc01309617e6d
I20260810 11:43:06.574558 2631 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:06.592612 2631 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:06.594441 2631 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:06.606393 2631 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccccb34fe1994d5588961cea0559bf97: Generated new TSK 0
I20260810 11:43:06.607280 2631 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:06.614393 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2508
I20260810 11:43:06.615537 2601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:06.713230 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2508
W20260810 11:43:06.741633 2487 connection.cc:535] client connection to 127.0.0.1:46643 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46643): Connection reset by peer (error 104)
2026-08-10T11:43:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2450 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:06 dist-test-slave-drl0 krb5kdc[2658](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:06 dist-test-slave-drl0 krb5kdc[2658](info): set up 2 sockets
Aug 10 11:43:06 dist-test-slave-drl0 krb5kdc[2658](info): commencing operation
krb5kdc: starting...
W20260810 11:43:08.101008 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:08 dist-test-slave-drl0 krb5kdc[2658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:08.463521 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43281
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43281
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:08.872217 2678 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:08.872588 2678 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:08.872767 2678 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:08.882216 2678 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:08.882398 2678 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:08.882513 2678 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:08.882635 2678 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:08.882720 2678 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:08.894331 2678 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43281
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:08.895746 2678 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:08.897507 2678 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:08.912266 2690 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:08.912293 2688 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:08.912218 2687 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:08.913033 2678 server_base.cc:1034] running on GCE node
I20260810 11:43:08.913888 2678 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:08.915437 2678 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:08.916682 2678 hybrid_clock.cc:648] HybridClock initialized: now 1786362188916640 us; error 32 us; skew 500 ppm
I20260810 11:43:08.917111 2678 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:08 dist-test-slave-drl0 krb5kdc[2658](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362188, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:08.921377 2678 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:08.923151 2678 webserver.cc:460] Webserver started at http://127.0.0.1:33083/ using document root <none> and password file <none>
I20260810 11:43:08.923888 2678 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:08.924044 2678 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:08.924377 2678 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:08.927536 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "ee6049fcce99457d8808b32a92d6c31e"
format_stamp: "Formatted at 2026-08-10 11:43:08 on dist-test-slave-drl0"
server_key: "88dd8685e5d90db15acf730ec788b0ea"
server_key_iv: "396cba6bffdfd73fc206d5989c4283b5"
server_key_version: "encryptionkey@0"
I20260810 11:43:08.928614 2678 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "ee6049fcce99457d8808b32a92d6c31e"
format_stamp: "Formatted at 2026-08-10 11:43:08 on dist-test-slave-drl0"
server_key: "88dd8685e5d90db15acf730ec788b0ea"
server_key_iv: "396cba6bffdfd73fc206d5989c4283b5"
server_key_version: "encryptionkey@0"
I20260810 11:43:08.936082 2678 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260810 11:43:08.940341 2701 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:08.941936 2678 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260810 11:43:08.942122 2678 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "ee6049fcce99457d8808b32a92d6c31e"
format_stamp: "Formatted at 2026-08-10 11:43:08 on dist-test-slave-drl0"
server_key: "88dd8685e5d90db15acf730ec788b0ea"
server_key_iv: "396cba6bffdfd73fc206d5989c4283b5"
server_key_version: "encryptionkey@0"
I20260810 11:43:08.942332 2678 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:08.967092 2678 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:08.972177 2678 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:08.972518 2678 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:08.995226 2678 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43281
I20260810 11:43:08.995469 2781 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43281 every 8 connection(s)
I20260810 11:43:08.996722 2678 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:09.000677 2784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:09.004282 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2678
I20260810 11:43:09.004508 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:09.005095 4035 external_mini_cluster.cc:1428] Setting key a2f7acafcff3279b70e55924eda29ac0
I20260810 11:43:09.014657 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: Bootstrap starting.
I20260810 11:43:09.018567 2784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:09.020006 2784 log.cc:824] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:09.021739 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:09.007300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:45735 (local address 127.0.0.1:43281)
0810 11:43:09.008071 (+ 771us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:09.008086 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:09.008138 (+ 52us) server_negotiation.cc:407] Connection header received
0810 11:43:09.010039 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:09.010072 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:09.010598 (+ 526us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:09.011056 (+ 458us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:09.011609 (+ 553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:09.013645 (+ 2036us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:09.015546 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:09.016944 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:09.017163 (+ 219us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:09.018858 (+ 1695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:09.018868 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:09.018887 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:09.019680 (+ 793us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:09.019788 (+ 108us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:09.019825 (+ 37us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:09.020007 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:09.020395 (+ 388us) server_negotiation.cc:299] Negotiation successful
0810 11:43:09.020587 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":331,"threads_started":1}
I20260810 11:43:09.023092 2784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: No bootstrap required, opened a new log
I20260810 11:43:09.027611 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:09.032059 2784 raft_consensus.cc:348] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260810 11:43:09.032577 2784 raft_consensus.cc:374] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:09.032711 2784 raft_consensus.cc:729] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee6049fcce99457d8808b32a92d6c31e, State: Initialized, Role: FOLLOWER
I20260810 11:43:09.033452 2784 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260810 11:43:09.033771 2784 raft_consensus.cc:388] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:09.033939 2784 raft_consensus.cc:482] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:09.034123 2784 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:09.036999 2784 raft_consensus.cc:504] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260810 11:43:09.037602 2784 leader_election.cc:302] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ee6049fcce99457d8808b32a92d6c31e; no voters:
I20260810 11:43:09.038357 2784 leader_election.cc:288] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:09.038498 2798 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:09.040051 2798 raft_consensus.cc:686] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [term 1 LEADER]: Becoming Leader. State: Replica: ee6049fcce99457d8808b32a92d6c31e, State: Running, Role: LEADER
I20260810 11:43:09.040856 2784 sys_catalog.cc:564] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:09.041016 2798 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } }
I20260810 11:43:09.041392 2788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:09.032683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:54585 (local address 127.0.0.1:43281)
0810 11:43:09.032864 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:09.032870 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:09.033581 (+ 711us) server_negotiation.cc:407] Connection header received
0810 11:43:09.033746 (+ 165us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:09.033754 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:09.033843 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:09.033982 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:09.034625 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:09.035864 (+ 1239us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:09.037193 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:09.037917 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:09.038083 (+ 166us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:09.040235 (+ 2152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:09.040248 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:09.040253 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:09.040289 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:09.040354 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:09.040380 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:09.040486 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:09.040965 (+ 479us) server_negotiation.cc:299] Negotiation successful
0810 11:43:09.041033 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260810 11:43:09.049993 2799 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ee6049fcce99457d8808b32a92d6c31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } } }
I20260810 11:43:09.050529 2799 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:09.051290 2801 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee6049fcce99457d8808b32a92d6c31e. Latest consensus state: current_term: 1 leader_uuid: "ee6049fcce99457d8808b32a92d6c31e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ee6049fcce99457d8808b32a92d6c31e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43281 } } }
I20260810 11:43:09.051589 2801 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e [sys.catalog]: This master's current role is: LEADER
W20260810 11:43:09.056162 2817 catalog_manager.cc:1558] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:43:09.056308 2817 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:43:09.056481 2819 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:09.058713 2819 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:09.068220 2819 catalog_manager.cc:1347] Generated new cluster ID: 64fe69cbe8114e94b2d6874906330477
I20260810 11:43:09.068360 2819 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:09.081881 2819 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:09.083700 2819 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:09.098059 2819 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ee6049fcce99457d8808b32a92d6c31e: Generated new TSK 0
I20260810 11:43:09.099136 2819 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:09.119035 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2678
W20260810 11:43:09.120362 2771 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.234:45735
W20260810 11:43:09.121229 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2678: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:09.121382 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2678
W20260810 11:43:09.149729 2656 connection.cc:535] client connection to 127.0.0.1:43281 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43281): Connection reset by peer (error 104)
2026-08-10T11:43:09Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2408 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:09 dist-test-slave-drl0 krb5kdc[2832](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:09 dist-test-slave-drl0 krb5kdc[2832](info): set up 2 sockets
Aug 10 11:43:09 dist-test-slave-drl0 krb5kdc[2832](info): commencing operation
krb5kdc: starting...
W20260810 11:43:10.518554 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.348s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:10 dist-test-slave-drl0 krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362190, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:10.911027 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35899
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35293
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:11.303993 2849 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:11.304381 2849 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:11.304585 2849 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:11.312708 2849 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:11.312902 2849 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:11.312968 2849 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:11.313035 2849 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:11.313086 2849 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:11.323612 2849 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35899
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35293
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35293
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:11.324908 2849 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:11.326237 2849 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:11.342280 2854 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:11.342432 2859 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:11.342515 2857 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:11.344229 2849 server_base.cc:1034] running on GCE node
I20260810 11:43:11.345124 2849 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:11.346714 2849 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:11.347972 2849 hybrid_clock.cc:648] HybridClock initialized: now 1786362191347912 us; error 45 us; skew 500 ppm
I20260810 11:43:11.348379 2849 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:11 dist-test-slave-drl0 krb5kdc[2832](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362191, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:11.353137 2849 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:11.355279 2849 webserver.cc:460] Webserver started at http://127.0.0.1:36141/ using document root <none> and password file <none>
I20260810 11:43:11.356137 2849 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:11.356277 2849 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:11.356601 2849 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:11.361512 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "c017ee38c00a44498c41da96262245bb"
format_stamp: "Formatted at 2026-08-10 11:43:11 on dist-test-slave-drl0"
server_key: "a4ff29022e147a8f8e87263941f70d28"
server_key_iv: "a97fc1a25628d99af51c9fa94c952563"
server_key_version: "encryptionkey@0"
I20260810 11:43:11.362416 2849 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "c017ee38c00a44498c41da96262245bb"
format_stamp: "Formatted at 2026-08-10 11:43:11 on dist-test-slave-drl0"
server_key: "a4ff29022e147a8f8e87263941f70d28"
server_key_iv: "a97fc1a25628d99af51c9fa94c952563"
server_key_version: "encryptionkey@0"
I20260810 11:43:11.368913 2849 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260810 11:43:11.372982 2869 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:11.374712 2849 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:43:11.374982 2849 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "c017ee38c00a44498c41da96262245bb"
format_stamp: "Formatted at 2026-08-10 11:43:11 on dist-test-slave-drl0"
server_key: "a4ff29022e147a8f8e87263941f70d28"
server_key_iv: "a97fc1a25628d99af51c9fa94c952563"
server_key_version: "encryptionkey@0"
I20260810 11:43:11.375205 2849 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:11.381601 2849 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:11.387279 2849 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:11.387653 2849 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:11.411785 2849 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35293
I20260810 11:43:11.411852 2943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35293 every 8 connection(s)
I20260810 11:43:11.413239 2849 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:11.415246 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 2849
I20260810 11:43:11.415458 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:11.416013 4035 external_mini_cluster.cc:1428] Setting key 8ed50328043e50a5a4ad0c136bdd2702
I20260810 11:43:11.418624 2944 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:11.424624 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:11.417893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:53103 (local address 127.0.0.1:35293)
0810 11:43:11.418982 (+ 1089us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:11.419002 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:11.419076 (+ 74us) server_negotiation.cc:407] Connection header received
0810 11:43:11.420969 (+ 1893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:11.420990 (+ 21us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:11.421368 (+ 378us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:11.421693 (+ 325us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:11.422796 (+ 1103us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:11.422811 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:11.422823 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:11.422841 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:11.422920 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:11.423001 (+ 81us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:11.423203 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:11.423303 (+ 100us) server_negotiation.cc:299] Negotiation successful
0810 11:43:11.423630 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":677,"thread_start_us":366,"threads_started":1}
I20260810 11:43:11.428952 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:11.431171 2944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb: Bootstrap starting.
I20260810 11:43:11.435338 2944 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:11.436474 2944 log.cc:824] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:11.437505 2947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:11.434345 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:53813 (local address 127.0.0.1:35293)
0810 11:43:11.434539 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:11.434544 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:11.435098 (+ 554us) server_negotiation.cc:407] Connection header received
0810 11:43:11.435248 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:11.435253 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:11.435325 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:11.435459 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:11.436398 (+ 939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:11.436404 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:11.436410 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:11.436442 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:11.436496 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:11.436521 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:11.436650 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:11.437011 (+ 361us) server_negotiation.cc:299] Negotiation successful
0810 11:43:11.437100 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:43:11.440723 2944 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb: No bootstrap required, opened a new log
I20260810 11:43:11.451159 2944 raft_consensus.cc:348] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260810 11:43:11.451557 2944 raft_consensus.cc:374] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:11.451659 2944 raft_consensus.cc:729] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c017ee38c00a44498c41da96262245bb, State: Initialized, Role: FOLLOWER
I20260810 11:43:11.452288 2944 consensus_queue.cc:260] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260810 11:43:11.452509 2944 raft_consensus.cc:388] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:11.452692 2944 raft_consensus.cc:482] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:11.452898 2944 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:11.455457 2944 raft_consensus.cc:504] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260810 11:43:11.456108 2944 leader_election.cc:302] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c017ee38c00a44498c41da96262245bb; no voters:
I20260810 11:43:11.456729 2944 leader_election.cc:288] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:11.456910 2955 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:11.457645 2955 raft_consensus.cc:686] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [term 1 LEADER]: Becoming Leader. State: Replica: c017ee38c00a44498c41da96262245bb, State: Running, Role: LEADER
I20260810 11:43:11.458223 2955 consensus_queue.cc:237] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } }
I20260810 11:43:11.459453 2944 sys_catalog.cc:564] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:11.461732 2959 sys_catalog.cc:455] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c017ee38c00a44498c41da96262245bb. Latest consensus state: current_term: 1 leader_uuid: "c017ee38c00a44498c41da96262245bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260810 11:43:11.462345 2959 sys_catalog.cc:458] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:11.462174 2957 sys_catalog.cc:455] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c017ee38c00a44498c41da96262245bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c017ee38c00a44498c41da96262245bb" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35293 } } }
I20260810 11:43:11.462575 2957 sys_catalog.cc:458] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:11.468277 2968 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:11.473727 2968 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:11.483484 2968 catalog_manager.cc:1347] Generated new cluster ID: 3d8fb77b6db44580ba0b52eebdb30c33
I20260810 11:43:11.483614 2968 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:11.517136 2968 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:11.517654 2968 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:11.535362 2968 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c017ee38c00a44498c41da96262245bb: Generated new TSK 0
I20260810 11:43:11.536204 2968 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:11.547242 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2849
W20260810 11:43:11.548659 2936 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.234:53103
W20260810 11:43:11.549383 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 2849: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:11.549538 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 2849
2026-08-10T11:43:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2425 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:11 dist-test-slave-drl0 krb5kdc[2992](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:11 dist-test-slave-drl0 krb5kdc[2992](info): set up 2 sockets
Aug 10 11:43:11 dist-test-slave-drl0 krb5kdc[2992](info): commencing operation
krb5kdc: starting...
W20260810 11:43:12.967654 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.373s user 0.003s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:13 dist-test-slave-drl0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:13.320813 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45037
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:13.698249 3017 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:13.698532 3017 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:13.698691 3017 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:13.706413 3017 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:13.706521 3017 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:13.706614 3017 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:13.706704 3017 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:13.706743 3017 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:13.717427 3017 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45037
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:13.719137 3017 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:13.720449 3017 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:13.732533 3022 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:13.732733 3025 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:13.732817 3023 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:13.733680 3017 server_base.cc:1034] running on GCE node
I20260810 11:43:13.735240 3017 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:13.737267 3017 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:13.738646 3017 hybrid_clock.cc:648] HybridClock initialized: now 1786362193738528 us; error 101 us; skew 500 ppm
I20260810 11:43:13.739137 3017 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:13 dist-test-slave-drl0 krb5kdc[2992](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362193, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:13.748600 3017 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:13.750881 3017 webserver.cc:460] Webserver started at http://127.0.0.1:39905/ using document root <none> and password file <none>
I20260810 11:43:13.751734 3017 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:13.751857 3017 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:13.752205 3017 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:13.755718 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c"
format_stamp: "Formatted at 2026-08-10 11:43:13 on dist-test-slave-drl0"
server_key: "7ce495966c5e97f5cefc1dfc0b8284a1"
server_key_iv: "c84841511ef59bf60059697df2041258"
server_key_version: "encryptionkey@0"
I20260810 11:43:13.756525 3017 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c"
format_stamp: "Formatted at 2026-08-10 11:43:13 on dist-test-slave-drl0"
server_key: "7ce495966c5e97f5cefc1dfc0b8284a1"
server_key_iv: "c84841511ef59bf60059697df2041258"
server_key_version: "encryptionkey@0"
I20260810 11:43:13.763343 3017 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260810 11:43:13.768587 3035 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:13.770298 3017 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260810 11:43:13.770642 3017 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c"
format_stamp: "Formatted at 2026-08-10 11:43:13 on dist-test-slave-drl0"
server_key: "7ce495966c5e97f5cefc1dfc0b8284a1"
server_key_iv: "c84841511ef59bf60059697df2041258"
server_key_version: "encryptionkey@0"
I20260810 11:43:13.770852 3017 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:13.800590 3017 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:13.805266 3017 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:13.805568 3017 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:13.828296 3017 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38781
I20260810 11:43:13.828274 3112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38781 every 8 connection(s)
I20260810 11:43:13.829811 3017 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:13.833658 3113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:13.838135 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3017
I20260810 11:43:13.838333 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:13.838977 4035 external_mini_cluster.cc:1428] Setting key 56cebfbc4674bddfe4d637d621a8ae8b
I20260810 11:43:13.847679 3113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c: Bootstrap starting.
I20260810 11:43:13.851229 3113 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:13.852448 3113 log.cc:824] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:43:13 dist-test-slave-drl0 krb5kdc[2992](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362193, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260810 11:43:13.855978 3113 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c: No bootstrap required, opened a new log
I20260810 11:43:13.864744 3113 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260810 11:43:13.865236 3113 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:13.865372 3113 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ee7b37afd6c4f28ab06ae0eb256e53c, State: Initialized, Role: FOLLOWER
I20260810 11:43:13.865994 3113 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260810 11:43:13.866242 3113 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:13.866466 3113 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:13.866657 3113 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:13.867138 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:13.841037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:33745 (local address 127.0.0.1:38781)
0810 11:43:13.841841 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:13.841853 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:13.841896 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:43:13.843805 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:13.843828 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:13.844693 (+ 865us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:13.845488 (+ 795us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:13.847583 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:13.849056 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:13.850493 (+ 1437us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:13.851228 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:13.851429 (+ 201us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:13.856905 (+ 5476us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:13.856913 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:13.856927 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:13.856968 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:13.861706 (+ 4738us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:13.862574 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:13.862580 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:13.862586 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:13.862765 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:13.863315 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:13.863318 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:13.863321 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:13.863881 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:13.864260 (+ 379us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:13.864773 (+ 513us) server_negotiation.cc:299] Negotiation successful
0810 11:43:13.865099 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":342,"threads_started":1}
I20260810 11:43:13.870311 3113 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260810 11:43:13.870996 3113 leader_election.cc:302] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ee7b37afd6c4f28ab06ae0eb256e53c; no voters:
I20260810 11:43:13.871687 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:13.871744 3113 leader_election.cc:288] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:13.871834 3124 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:13.874356 3113 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:13.875380 3124 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [term 1 LEADER]: Becoming Leader. State: Replica: 9ee7b37afd6c4f28ab06ae0eb256e53c, State: Running, Role: LEADER
I20260810 11:43:13.876199 3124 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20260810 11:43:13.879670 3127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ee7b37afd6c4f28ab06ae0eb256e53c. Latest consensus state: current_term: 1 leader_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20260810 11:43:13.880142 3127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:13.880817 3125 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ee7b37afd6c4f28ab06ae0eb256e53c" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20260810 11:43:13.881194 3125 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:13.896102 3138 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:13.903079 3138 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:13.933975 3138 catalog_manager.cc:1347] Generated new cluster ID: 8d5be86d74ee476cb20c0f9a97165dc2
I20260810 11:43:13.934104 3138 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:13.940996 3120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:13.891414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:35309 (local address 127.0.0.1:38781)
0810 11:43:13.891651 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:13.891661 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:13.911829 (+ 20168us) server_negotiation.cc:407] Connection header received
0810 11:43:13.912180 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:13.912186 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:13.912275 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:13.912420 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:13.915138 (+ 2718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:13.916707 (+ 1569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:13.919236 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:13.920139 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:13.920267 (+ 128us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:13.925121 (+ 4854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:13.925145 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:13.925153 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:13.925214 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:13.936806 (+ 11592us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:13.937851 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:13.937857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:13.937861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:13.937994 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:13.938557 (+ 563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:13.938562 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:13.938566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:13.938990 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:13.939239 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:13.940422 (+ 1183us) server_negotiation.cc:299] Negotiation successful
0810 11:43:13.940598 (+ 176us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260810 11:43:13.953475 3138 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:13.954856 3138 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:13.961573 3138 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ee7b37afd6c4f28ab06ae0eb256e53c: Generated new TSK 0
I20260810 11:43:13.962365 3138 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:13.969049 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3017
I20260810 11:43:13.970923 3107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:14.066087 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3017
W20260810 11:43:14.093825 2988 connection.cc:535] client connection to 127.0.0.1:38781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38781): Connection reset by peer (error 104)
2026-08-10T11:43:14Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2519 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:14 dist-test-slave-drl0 krb5kdc[3165](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:14 dist-test-slave-drl0 krb5kdc[3165](info): set up 2 sockets
Aug 10 11:43:14 dist-test-slave-drl0 krb5kdc[3165](info): commencing operation
krb5kdc: starting...
W20260810 11:43:15.444823 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.003s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:15 dist-test-slave-drl0 krb5kdc[3165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362195, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:15.795315 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44251
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:16.181144 3185 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:16.181455 3185 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:16.181640 3185 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:16.190176 3185 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:16.190284 3185 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:16.190446 3185 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:16.190533 3185 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:16.190620 3185 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:16.200587 3185 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44251
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45739
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:16.201856 3185 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:16.203562 3185 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:16.216490 3194 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:16.216588 3190 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:16.216552 3192 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:16.217886 3185 server_base.cc:1034] running on GCE node
I20260810 11:43:16.218650 3185 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:16.221196 3185 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:16.222565 3185 hybrid_clock.cc:648] HybridClock initialized: now 1786362196222486 us; error 68 us; skew 500 ppm
I20260810 11:43:16.223070 3185 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:16 dist-test-slave-drl0 krb5kdc[3165](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362196, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:16.227746 3185 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:16.229936 3185 webserver.cc:460] Webserver started at http://127.0.0.1:43729/ using document root <none> and password file <none>
I20260810 11:43:16.230907 3185 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:16.231186 3185 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:16.231608 3185 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:16.235479 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "2c6bf48f45f34f9588b3791a479daa92"
format_stamp: "Formatted at 2026-08-10 11:43:16 on dist-test-slave-drl0"
server_key: "7e1596426e287d15a48b92ccb1eeb90b"
server_key_iv: "805777979cab5e857b9a16bec55b5424"
server_key_version: "encryptionkey@0"
I20260810 11:43:16.236369 3185 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "2c6bf48f45f34f9588b3791a479daa92"
format_stamp: "Formatted at 2026-08-10 11:43:16 on dist-test-slave-drl0"
server_key: "7e1596426e287d15a48b92ccb1eeb90b"
server_key_iv: "805777979cab5e857b9a16bec55b5424"
server_key_version: "encryptionkey@0"
I20260810 11:43:16.243602 3185 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.000s sys 0.007s
I20260810 11:43:16.248040 3201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:16.249895 3185 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:43:16.250110 3185 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "2c6bf48f45f34f9588b3791a479daa92"
format_stamp: "Formatted at 2026-08-10 11:43:16 on dist-test-slave-drl0"
server_key: "7e1596426e287d15a48b92ccb1eeb90b"
server_key_iv: "805777979cab5e857b9a16bec55b5424"
server_key_version: "encryptionkey@0"
I20260810 11:43:16.250334 3185 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:16.289196 3185 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:16.293965 3185 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:16.294287 3185 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:16.317579 3185 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45739
I20260810 11:43:16.317612 3278 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:45739 every 8 connection(s)
I20260810 11:43:16.319034 3185 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:16.320694 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3185
I20260810 11:43:16.320914 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:16.321391 4035 external_mini_cluster.cc:1428] Setting key 543fbc684402573f8ea1b8e69bc49321
I20260810 11:43:16.323478 3279 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:16.335678 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:16.323241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:55343 (local address 127.0.0.1:45739)
0810 11:43:16.324059 (+ 818us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:16.324071 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:16.324112 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:43:16.326051 (+ 1939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:16.326078 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:16.326450 (+ 372us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:16.326806 (+ 356us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:16.327523 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:16.328879 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:16.330894 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:16.331625 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:16.331768 (+ 143us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:16.333236 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:16.333243 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:16.333256 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:16.333811 (+ 555us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:16.333891 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:16.333919 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:16.334052 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:16.334471 (+ 419us) server_negotiation.cc:299] Negotiation successful
0810 11:43:16.334662 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":613,"thread_start_us":375,"threads_started":1}
I20260810 11:43:16.337041 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92: Bootstrap starting.
I20260810 11:43:16.340595 3279 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:16.340792 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:16.341710 3279 log.cc:824] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:16.344929 3279 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92: No bootstrap required, opened a new log
I20260810 11:43:16.352946 3286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:16.345817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:56405 (local address 127.0.0.1:45739)
0810 11:43:16.345971 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:16.345975 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:16.346497 (+ 522us) server_negotiation.cc:407] Connection header received
0810 11:43:16.346729 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:16.346733 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:16.346795 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:16.346972 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:16.347544 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:16.348367 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:16.349709 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:16.350185 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:16.350659 (+ 474us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:16.352063 (+ 1404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:16.352071 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:16.352077 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:16.352098 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:16.352163 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:16.352185 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:16.352328 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:16.352536 (+ 208us) server_negotiation.cc:299] Negotiation successful
0810 11:43:16.352596 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20260810 11:43:16.353775 3279 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260810 11:43:16.354173 3279 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:16.354305 3279 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c6bf48f45f34f9588b3791a479daa92, State: Initialized, Role: FOLLOWER
I20260810 11:43:16.354992 3279 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260810 11:43:16.355237 3279 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:16.355391 3279 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:16.355602 3279 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:16.357796 3279 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260810 11:43:16.358494 3279 leader_election.cc:302] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2c6bf48f45f34f9588b3791a479daa92; no voters:
I20260810 11:43:16.359187 3279 leader_election.cc:288] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:16.359354 3294 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:16.360185 3294 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [term 1 LEADER]: Becoming Leader. State: Replica: 2c6bf48f45f34f9588b3791a479daa92, State: Running, Role: LEADER
I20260810 11:43:16.360739 3294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } }
I20260810 11:43:16.361418 3279 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:16.367528 3296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2c6bf48f45f34f9588b3791a479daa92. Latest consensus state: current_term: 1 leader_uuid: "2c6bf48f45f34f9588b3791a479daa92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260810 11:43:16.368937 3296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:16.370971 3295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2c6bf48f45f34f9588b3791a479daa92" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2c6bf48f45f34f9588b3791a479daa92" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45739 } } }
I20260810 11:43:16.371546 3306 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:16.375350 3295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:16.379302 3306 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:16.388967 3306 catalog_manager.cc:1347] Generated new cluster ID: d0acb351b2ff4e3eb5d77e543fce598e
I20260810 11:43:16.389084 3306 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:16.406169 3306 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:16.407742 3306 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:16.415930 3306 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2c6bf48f45f34f9588b3791a479daa92: Generated new TSK 0
I20260810 11:43:16.416879 3306 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:16.429963 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3185
W20260810 11:43:16.432190 3272 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.234:55343
W20260810 11:43:16.433313 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3185: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:16.433457 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3185
W20260810 11:43:16.462689 3161 connection.cc:535] client connection to 127.0.0.1:45739 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:45739): Connection reset by peer (error 104)
2026-08-10T11:43:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2368 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:16 dist-test-slave-drl0 krb5kdc[3319](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:16 dist-test-slave-drl0 krb5kdc[3319](info): set up 2 sockets
Aug 10 11:43:16 dist-test-slave-drl0 krb5kdc[3319](info): commencing operation
krb5kdc: starting...
W20260810 11:43:17.823526 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.006s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:18 dist-test-slave-drl0 krb5kdc[3319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362198, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260810 11:43:18.187677 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43509
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:18.574564 3338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:18.575057 3338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:18.575460 3338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:18.588689 3338 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:18.588871 3338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:18.588971 3338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:18.589080 3338 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:18.589157 3338 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:18.600482 3338 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43509
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43509
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:18.601967 3338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:18.603500 3338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:18.616043 3352 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:18.616078 3349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:18.616384 3347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:18.617233 3338 server_base.cc:1034] running on GCE node
I20260810 11:43:18.617942 3338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:18.619349 3338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:18.620568 3338 hybrid_clock.cc:648] HybridClock initialized: now 1786362198620507 us; error 47 us; skew 500 ppm
I20260810 11:43:18.621040 3338 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:18 dist-test-slave-drl0 krb5kdc[3319](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362198, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:18.625994 3338 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260810 11:43:18.628175 3338 webserver.cc:460] Webserver started at http://127.0.0.1:40263/ using document root <none> and password file <none>
I20260810 11:43:18.629081 3338 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:18.629238 3338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:18.629689 3338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:18.632987 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "6fcc7c07892d4ace94c5654d0c46149d"
format_stamp: "Formatted at 2026-08-10 11:43:18 on dist-test-slave-drl0"
server_key: "730ced3ad00c904943f8c0eae0d5b068"
server_key_iv: "462ae339c191fa674576964b2b8b7d42"
server_key_version: "encryptionkey@0"
I20260810 11:43:18.633738 3338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "6fcc7c07892d4ace94c5654d0c46149d"
format_stamp: "Formatted at 2026-08-10 11:43:18 on dist-test-slave-drl0"
server_key: "730ced3ad00c904943f8c0eae0d5b068"
server_key_iv: "462ae339c191fa674576964b2b8b7d42"
server_key_version: "encryptionkey@0"
I20260810 11:43:18.639653 3338 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260810 11:43:18.643360 3360 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:18.645298 3338 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260810 11:43:18.645730 3338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "6fcc7c07892d4ace94c5654d0c46149d"
format_stamp: "Formatted at 2026-08-10 11:43:18 on dist-test-slave-drl0"
server_key: "730ced3ad00c904943f8c0eae0d5b068"
server_key_iv: "462ae339c191fa674576964b2b8b7d42"
server_key_version: "encryptionkey@0"
I20260810 11:43:18.646013 3338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:18.653296 3338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:18.657981 3338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:18.658466 3338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:18.683102 3338 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43509
I20260810 11:43:18.683194 3437 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43509 every 8 connection(s)
I20260810 11:43:18.684551 3338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:18.688378 3438 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:18.692360 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3338
I20260810 11:43:18.692544 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:18.693023 4035 external_mini_cluster.cc:1428] Setting key 5926c710fa26ba6369d2eac0caff9a42
I20260810 11:43:18.702279 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:18.694906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:43947 (local address 127.0.0.1:43509)
0810 11:43:18.695706 (+ 800us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:18.695720 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:18.695772 (+ 52us) server_negotiation.cc:407] Connection header received
0810 11:43:18.697622 (+ 1850us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:18.697655 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:18.698305 (+ 650us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:18.698778 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:18.700242 (+ 1464us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:18.700252 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:18.700269 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:18.700293 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:18.700396 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:18.700438 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:18.700675 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:18.700822 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 11:43:18.701028 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":302,"threads_started":1}
I20260810 11:43:18.702764 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d: Bootstrap starting.
I20260810 11:43:18.706776 3438 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:18.707675 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:18.708240 3438 log.cc:824] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:18.711745 3438 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d: No bootstrap required, opened a new log
I20260810 11:43:18.716332 3441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:18.713507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.234:53499 (local address 127.0.0.1:43509)
0810 11:43:18.713691 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:18.713697 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:18.713975 (+ 278us) server_negotiation.cc:407] Connection header received
0810 11:43:18.714101 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:18.714107 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:18.714164 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:18.714382 (+ 218us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:18.715556 (+ 1174us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:18.715562 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:18.715567 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:18.715586 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:18.715638 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:18.715663 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:18.715807 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:18.715975 (+ 168us) server_negotiation.cc:299] Negotiation successful
0810 11:43:18.716044 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:43:18.722307 3438 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260810 11:43:18.722710 3438 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:18.722827 3438 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6fcc7c07892d4ace94c5654d0c46149d, State: Initialized, Role: FOLLOWER
I20260810 11:43:18.723434 3438 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260810 11:43:18.723706 3438 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:18.723881 3438 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:18.724092 3438 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:18.726920 3438 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260810 11:43:18.727460 3438 leader_election.cc:302] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6fcc7c07892d4ace94c5654d0c46149d; no voters:
I20260810 11:43:18.728035 3438 leader_election.cc:288] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:18.728325 3452 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:18.729192 3452 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [term 1 LEADER]: Becoming Leader. State: Replica: 6fcc7c07892d4ace94c5654d0c46149d, State: Running, Role: LEADER
I20260810 11:43:18.729733 3452 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } }
I20260810 11:43:18.730283 3438 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:18.732612 3453 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6fcc7c07892d4ace94c5654d0c46149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260810 11:43:18.732586 3454 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6fcc7c07892d4ace94c5654d0c46149d. Latest consensus state: current_term: 1 leader_uuid: "6fcc7c07892d4ace94c5654d0c46149d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6fcc7c07892d4ace94c5654d0c46149d" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43509 } } }
I20260810 11:43:18.733418 3453 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:18.733448 3454 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:18.736975 3461 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:18.741636 3461 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:18.751362 3461 catalog_manager.cc:1347] Generated new cluster ID: e779a513633842df91c6ed0c132ebcab
I20260810 11:43:18.751498 3461 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:18.774601 3461 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:18.775107 3461 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:18.787710 3461 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6fcc7c07892d4ace94c5654d0c46149d: Generated new TSK 0
I20260810 11:43:18.788841 3461 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:18.801451 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3338
W20260810 11:43:18.802904 3430 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.234:43947
W20260810 11:43:18.803922 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3338: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:18.804134 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3338
2026-08-10T11:43:18Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2369 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:18 dist-test-slave-drl0 krb5kdc[3481](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:18 dist-test-slave-drl0 krb5kdc[3481](info): set up 2 sockets
Aug 10 11:43:18 dist-test-slave-drl0 krb5kdc[3481](info): commencing operation
krb5kdc: starting...
W20260810 11:43:20.199677 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.347s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:20 dist-test-slave-drl0 krb5kdc[3481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:20Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:20.570508 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44265
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:40479
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44265
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:20.953524 3503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:20.953823 3503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:20.953981 3503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:20.962800 3503 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:20.962998 3503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:20.963104 3503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:20.963317 3503 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:20.963402 3503 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:20.976192 3503 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:40479
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44265
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:20.977809 3503 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:20.979411 3503 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:20.995412 3519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:20.995393 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:20.995689 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:20.996137 3503 server_base.cc:1034] running on GCE node
I20260810 11:43:20.996975 3503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:20.998412 3503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:20.999737 3503 hybrid_clock.cc:648] HybridClock initialized: now 1786362200999690 us; error 67 us; skew 500 ppm
I20260810 11:43:21.000133 3503 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:20 dist-test-slave-drl0 krb5kdc[3481](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362200, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:21.003391 3503 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:21.004956 3503 webserver.cc:460] Webserver started at http://127.3.240.254:41551/ using document root <none> and password file <none>
I20260810 11:43:21.005587 3503 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:21.005692 3503 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:21.005944 3503 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:21.009500 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "41a8f27dbb6140eeb1901d447a67a447"
format_stamp: "Formatted at 2026-08-10 11:43:21 on dist-test-slave-drl0"
server_key: "f519a713669599b2940fa5ac9881ae60"
server_key_iv: "e8db68d7e912b88669132a6cbd9c5be1"
server_key_version: "encryptionkey@0"
I20260810 11:43:21.010246 3503 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "41a8f27dbb6140eeb1901d447a67a447"
format_stamp: "Formatted at 2026-08-10 11:43:21 on dist-test-slave-drl0"
server_key: "f519a713669599b2940fa5ac9881ae60"
server_key_iv: "e8db68d7e912b88669132a6cbd9c5be1"
server_key_version: "encryptionkey@0"
I20260810 11:43:21.015841 3503 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:43:21.020037 3529 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:21.021765 3503 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:43:21.021970 3503 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "41a8f27dbb6140eeb1901d447a67a447"
format_stamp: "Formatted at 2026-08-10 11:43:21 on dist-test-slave-drl0"
server_key: "f519a713669599b2940fa5ac9881ae60"
server_key_iv: "e8db68d7e912b88669132a6cbd9c5be1"
server_key_version: "encryptionkey@0"
I20260810 11:43:21.022150 3503 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:21.040714 3503 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:21.045970 3503 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:21.046279 3503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:21.069859 3503 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44265
I20260810 11:43:21.069885 3603 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44265 every 8 connection(s)
I20260810 11:43:21.071312 3503 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:21.075107 3604 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:21.078668 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3503
I20260810 11:43:21.079082 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:21.080214 4035 external_mini_cluster.cc:1428] Setting key df338d394cbfb398be258f86b2ab844a
I20260810 11:43:21.089900 3604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447: Bootstrap starting.
I20260810 11:43:21.093621 3604 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:21.094648 3604 log.cc:824] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447: Log is configured to *not* fsync() on all Append() calls
Aug 10 11:43:21 dist-test-slave-drl0 krb5kdc[3481](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362200, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:21.098004 3604 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447: No bootstrap required, opened a new log
I20260810 11:43:21.108857 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:21.082445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48379 (local address 127.3.240.254:44265)
0810 11:43:21.083786 (+ 1341us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:21.083802 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:21.083858 (+ 56us) server_negotiation.cc:407] Connection header received
0810 11:43:21.085688 (+ 1830us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:21.085718 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:21.086235 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:21.086672 (+ 437us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:21.088179 (+ 1507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:21.090133 (+ 1954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:21.091520 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:21.092555 (+ 1035us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:21.092743 (+ 188us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:21.097563 (+ 4820us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:21.097575 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:21.097593 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:21.097637 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:21.103212 (+ 5575us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:21.104648 (+ 1436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:21.104661 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:21.104669 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:21.104901 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:21.105497 (+ 596us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:21.105502 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:21.105506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:21.106248 (+ 742us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:21.106604 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:21.107079 (+ 475us) server_negotiation.cc:299] Negotiation successful
0810 11:43:21.107428 (+ 349us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":928,"thread_start_us":584,"threads_started":1}
I20260810 11:43:21.110045 3604 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } }
I20260810 11:43:21.110517 3604 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:21.110653 3604 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41a8f27dbb6140eeb1901d447a67a447, State: Initialized, Role: FOLLOWER
I20260810 11:43:21.111578 3604 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } }
I20260810 11:43:21.111937 3604 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:21.112094 3604 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:21.112277 3604 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:21.113299 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:21.114715 3604 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } }
I20260810 11:43:21.115255 3604 leader_election.cc:302] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41a8f27dbb6140eeb1901d447a67a447; no voters:
I20260810 11:43:21.115926 3604 leader_election.cc:288] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:21.116039 3609 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:21.116801 3609 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [term 1 LEADER]: Becoming Leader. State: Replica: 41a8f27dbb6140eeb1901d447a67a447, State: Running, Role: LEADER
I20260810 11:43:21.117589 3609 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } }
I20260810 11:43:21.119773 3604 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:21.121711 3615 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41a8f27dbb6140eeb1901d447a67a447. Latest consensus state: current_term: 1 leader_uuid: "41a8f27dbb6140eeb1901d447a67a447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } } }
I20260810 11:43:21.122160 3615 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:21.122179 3618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41a8f27dbb6140eeb1901d447a67a447" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41a8f27dbb6140eeb1901d447a67a447" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44265 } } }
I20260810 11:43:21.122972 3618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:21.125334 3625 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:21.131062 3625 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:21.147243 3607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:21.121837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48837 (local address 127.3.240.254:44265)
0810 11:43:21.122338 (+ 501us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:21.122345 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:21.122368 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:43:21.122578 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:21.122589 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:21.122725 (+ 136us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:21.122918 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:21.125447 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:21.126664 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:21.128835 (+ 2171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:21.129829 (+ 994us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:21.129997 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:21.134321 (+ 4324us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:21.134332 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:21.134339 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:21.134391 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:21.140414 (+ 6023us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:21.141979 (+ 1565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:21.141988 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:21.141994 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:21.142198 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:21.143105 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:21.143114 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:21.143120 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:21.143974 (+ 854us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:21.144696 (+ 722us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:21.146426 (+ 1730us) server_negotiation.cc:299] Negotiation successful
0810 11:43:21.146673 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":361}
I20260810 11:43:21.153527 3625 catalog_manager.cc:1347] Generated new cluster ID: 81814320ea724a87939e829eef333c62
I20260810 11:43:21.153647 3625 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:21.177059 3625 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:21.178556 3625 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:21.188792 3625 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41a8f27dbb6140eeb1901d447a67a447: Generated new TSK 0
I20260810 11:43:21.190359 3625 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:21.202507 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3503
I20260810 11:43:21.204058 3598 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:21.305574 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3503
W20260810 11:43:21.335647 3478 connection.cc:535] client connection to 127.3.240.254:44265 recv error: Network error: failed to read from TLS socket (remote: 127.3.240.254:44265): Connection reset by peer (error 104)
2026-08-10T11:43:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2503 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:21 dist-test-slave-drl0 krb5kdc[3649](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:21 dist-test-slave-drl0 krb5kdc[3649](info): set up 2 sockets
Aug 10 11:43:21 dist-test-slave-drl0 krb5kdc[3649](info): commencing operation
krb5kdc: starting...
W20260810 11:43:22.694073 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.001s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:22 dist-test-slave-drl0 krb5kdc[3649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362202, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:23.066732 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44225
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:35359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44225
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:23.452190 3670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:23.452716 3670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:23.453097 3670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:23.463117 3670 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:23.463310 3670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:23.463418 3670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:23.463538 3670 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:23.463632 3670 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:23.475682 3670 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:35359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44225
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44225
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:23.477154 3670 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:23.478508 3670 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:23.491535 3676 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:23.491602 3677 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:23.491535 3679 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:23.493824 3670 server_base.cc:1034] running on GCE node
I20260810 11:43:23.494658 3670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:23.496851 3670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:23.498126 3670 hybrid_clock.cc:648] HybridClock initialized: now 1786362203498066 us; error 51 us; skew 500 ppm
I20260810 11:43:23.498545 3670 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:23 dist-test-slave-drl0 krb5kdc[3649](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362203, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:23.502801 3670 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:23.504673 3670 webserver.cc:460] Webserver started at http://127.3.240.254:38569/ using document root <none> and password file <none>
I20260810 11:43:23.505405 3670 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:23.505563 3670 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:23.505896 3670 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:23.509219 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "6cca5b17b76c4333a25ea61b19f1a301"
format_stamp: "Formatted at 2026-08-10 11:43:23 on dist-test-slave-drl0"
server_key: "21f6704f326e197fd8dc5cb123a6a4a5"
server_key_iv: "14625d68ea1bc0a6fd9b87f012c1fee7"
server_key_version: "encryptionkey@0"
I20260810 11:43:23.510160 3670 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "6cca5b17b76c4333a25ea61b19f1a301"
format_stamp: "Formatted at 2026-08-10 11:43:23 on dist-test-slave-drl0"
server_key: "21f6704f326e197fd8dc5cb123a6a4a5"
server_key_iv: "14625d68ea1bc0a6fd9b87f012c1fee7"
server_key_version: "encryptionkey@0"
I20260810 11:43:23.516355 3670 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260810 11:43:23.520655 3687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:23.522753 3670 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260810 11:43:23.523043 3670 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "6cca5b17b76c4333a25ea61b19f1a301"
format_stamp: "Formatted at 2026-08-10 11:43:23 on dist-test-slave-drl0"
server_key: "21f6704f326e197fd8dc5cb123a6a4a5"
server_key_iv: "14625d68ea1bc0a6fd9b87f012c1fee7"
server_key_version: "encryptionkey@0"
I20260810 11:43:23.523296 3670 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:23.549101 3670 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:23.554602 3670 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:23.554979 3670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:23.578038 3670 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44225
I20260810 11:43:23.578069 3768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44225 every 8 connection(s)
I20260810 11:43:23.579504 3670 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:23.583847 3769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:23.586655 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3670
I20260810 11:43:23.586879 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:23.587477 4035 external_mini_cluster.cc:1428] Setting key 0bdc5a6518443355f2f6769b098c8e8f
I20260810 11:43:23.596979 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301: Bootstrap starting.
I20260810 11:43:23.600719 3769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:23.601915 3769 log.cc:824] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:23.602581 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:23.589446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53535 (local address 127.3.240.254:44225)
0810 11:43:23.590327 (+ 881us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:23.590338 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:23.590380 (+ 42us) server_negotiation.cc:407] Connection header received
0810 11:43:23.592388 (+ 2008us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:23.592415 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:23.592810 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:23.593153 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:23.593819 (+ 666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:23.595310 (+ 1491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:23.597333 (+ 2023us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:23.598384 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:23.598714 (+ 330us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:23.600083 (+ 1369us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:23.600091 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:23.600103 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:23.600641 (+ 538us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:23.600720 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:23.600746 (+ 26us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:23.600983 (+ 237us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:23.601347 (+ 364us) server_negotiation.cc:299] Negotiation successful
0810 11:43:23.601522 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":415,"threads_started":1}
I20260810 11:43:23.606053 3769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301: No bootstrap required, opened a new log
I20260810 11:43:23.606838 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:23.614678 3769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } }
I20260810 11:43:23.615204 3769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:23.615386 3769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cca5b17b76c4333a25ea61b19f1a301, State: Initialized, Role: FOLLOWER
I20260810 11:43:23.615983 3769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } }
I20260810 11:43:23.616217 3769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:23.616365 3769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:23.616562 3769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:23.619867 3769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } }
I20260810 11:43:23.620548 3769 leader_election.cc:302] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6cca5b17b76c4333a25ea61b19f1a301; no voters:
I20260810 11:43:23.620823 3773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:23.611473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55413 (local address 127.3.240.254:44225)
0810 11:43:23.611671 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:23.611677 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:23.612258 (+ 581us) server_negotiation.cc:407] Connection header received
0810 11:43:23.612420 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:23.612430 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:23.612503 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:23.612633 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:23.613773 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:23.615097 (+ 1324us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:23.617411 (+ 2314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:23.618141 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:23.618300 (+ 159us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:23.619665 (+ 1365us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:23.619673 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:23.619681 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:23.619703 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:23.619891 (+ 188us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:23.619932 (+ 41us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:23.620078 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:23.620389 (+ 311us) server_negotiation.cc:299] Negotiation successful
0810 11:43:23.620444 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:43:23.621439 3769 leader_election.cc:288] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:23.621840 3781 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:23.623534 3781 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [term 1 LEADER]: Becoming Leader. State: Replica: 6cca5b17b76c4333a25ea61b19f1a301, State: Running, Role: LEADER
I20260810 11:43:23.624117 3781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } }
I20260810 11:43:23.625003 3769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:23.627262 3782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6cca5b17b76c4333a25ea61b19f1a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } } }
I20260810 11:43:23.627871 3782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:23.627393 3783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6cca5b17b76c4333a25ea61b19f1a301. Latest consensus state: current_term: 1 leader_uuid: "6cca5b17b76c4333a25ea61b19f1a301" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6cca5b17b76c4333a25ea61b19f1a301" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44225 } } }
I20260810 11:43:23.628324 3783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:23.632722 3788 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:23.642697 3788 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:23.652968 3788 catalog_manager.cc:1347] Generated new cluster ID: e654447094c0437a84c186e39f73a2c4
I20260810 11:43:23.653091 3788 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:23.678387 3788 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:23.680348 3788 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:23.695984 3788 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6cca5b17b76c4333a25ea61b19f1a301: Generated new TSK 0
I20260810 11:43:23.696877 3788 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:23.704541 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3670
W20260810 11:43:23.705701 3760 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:53535
W20260810 11:43:23.706580 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3670: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:23.707022 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3670
W20260810 11:43:23.735173 3647 connection.cc:535] client connection to 127.3.240.254:44225 recv error: Network error: failed to read from TLS socket (remote: 127.3.240.254:44225): Connection reset by peer (error 104)
2026-08-10T11:43:23Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2400 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:23 dist-test-slave-drl0 krb5kdc[3807](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:23 dist-test-slave-drl0 krb5kdc[3807](info): set up 2 sockets
Aug 10 11:43:23 dist-test-slave-drl0 krb5kdc[3807](info): commencing operation
krb5kdc: starting...
W20260810 11:43:25.119692 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.363s user 0.004s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:25 dist-test-slave-drl0 krb5kdc[3807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362205, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:25Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:25Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:25.488044 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44505
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:39731
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:44505
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:25.857782 3825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:25.858119 3825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:25.858299 3825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:25.865935 3825 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:25.866108 3825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:25.866204 3825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:25.866298 3825 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:25.866390 3825 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:25.876955 3825 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:39731
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:44505
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:44505
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:25.878312 3825 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:25.879868 3825 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:25.893756 3833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:25.893697 3837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:25.893801 3834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:25.895175 3825 server_base.cc:1034] running on GCE node
I20260810 11:43:25.895923 3825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:25.897665 3825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:25.898978 3825 hybrid_clock.cc:648] HybridClock initialized: now 1786362205898834 us; error 81 us; skew 500 ppm
I20260810 11:43:25.899379 3825 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:25 dist-test-slave-drl0 krb5kdc[3807](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362205, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:25.902804 3825 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:25.904686 3825 webserver.cc:460] Webserver started at http://127.3.240.254:34083/ using document root <none> and password file <none>
I20260810 11:43:25.905400 3825 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:25.905553 3825 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:25.905871 3825 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:25.908946 3825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "7e213916ed404f3e932b6fb7c9ed2c65"
format_stamp: "Formatted at 2026-08-10 11:43:25 on dist-test-slave-drl0"
server_key: "ab9610c3e7960111edab4c5816528573"
server_key_iv: "476b19ab13ace5824eb5ad66605d7b04"
server_key_version: "encryptionkey@0"
I20260810 11:43:25.909740 3825 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "7e213916ed404f3e932b6fb7c9ed2c65"
format_stamp: "Formatted at 2026-08-10 11:43:25 on dist-test-slave-drl0"
server_key: "ab9610c3e7960111edab4c5816528573"
server_key_iv: "476b19ab13ace5824eb5ad66605d7b04"
server_key_version: "encryptionkey@0"
I20260810 11:43:25.915509 3825 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260810 11:43:25.919785 3846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:25.921195 3825 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260810 11:43:25.921404 3825 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "7e213916ed404f3e932b6fb7c9ed2c65"
format_stamp: "Formatted at 2026-08-10 11:43:25 on dist-test-slave-drl0"
server_key: "ab9610c3e7960111edab4c5816528573"
server_key_iv: "476b19ab13ace5824eb5ad66605d7b04"
server_key_version: "encryptionkey@0"
I20260810 11:43:25.921622 3825 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:25.928164 3825 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:25.933216 3825 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:25.933535 3825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:25.955862 3825 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:44505
I20260810 11:43:25.955859 3933 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:44505 every 8 connection(s)
I20260810 11:43:25.957341 3825 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:25.961299 3934 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:25.964331 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 3825
I20260810 11:43:25.964531 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:25.965107 4035 external_mini_cluster.cc:1428] Setting key 81bc3ae9cdbc2b3bc78166723c78af59
I20260810 11:43:25.973788 3934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65: Bootstrap starting.
I20260810 11:43:25.974551 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:25.967165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38101 (local address 127.3.240.254:44505)
0810 11:43:25.968100 (+ 935us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:25.968112 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:25.968153 (+ 41us) server_negotiation.cc:407] Connection header received
0810 11:43:25.969954 (+ 1801us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:25.969982 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:25.970744 (+ 762us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:25.971501 (+ 757us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:25.972911 (+ 1410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:25.972917 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:25.972931 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:25.972956 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:25.973009 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:25.973029 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:25.973275 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:25.973430 (+ 155us) server_negotiation.cc:299] Negotiation successful
0810 11:43:25.973610 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":726,"thread_start_us":475,"threads_started":1}
I20260810 11:43:25.977046 3934 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:25.978197 3934 log.cc:824] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:25.978561 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:25.981076 3934 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65: No bootstrap required, opened a new log
I20260810 11:43:25.986703 3942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:25.984034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38007 (local address 127.3.240.254:44505)
0810 11:43:25.984235 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:25.984241 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:25.984643 (+ 402us) server_negotiation.cc:407] Connection header received
0810 11:43:25.984769 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:25.984773 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:25.984833 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:25.984938 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:25.985943 (+ 1005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:25.985954 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:25.985959 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:25.985978 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:25.986026 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:25.986046 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:25.986175 (+ 129us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:25.986328 (+ 153us) server_negotiation.cc:299] Negotiation successful
0810 11:43:25.986388 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20260810 11:43:25.989565 3934 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } }
I20260810 11:43:25.989914 3934 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:25.990015 3934 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e213916ed404f3e932b6fb7c9ed2c65, State: Initialized, Role: FOLLOWER
I20260810 11:43:25.990590 3934 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } }
I20260810 11:43:25.990839 3934 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:25.991015 3934 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:25.991192 3934 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:25.993729 3934 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } }
I20260810 11:43:25.994230 3934 leader_election.cc:302] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7e213916ed404f3e932b6fb7c9ed2c65; no voters:
I20260810 11:43:25.995146 3934 leader_election.cc:288] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:25.995361 3950 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:25.996049 3950 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [term 1 LEADER]: Becoming Leader. State: Replica: 7e213916ed404f3e932b6fb7c9ed2c65, State: Running, Role: LEADER
I20260810 11:43:25.996610 3950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } }
I20260810 11:43:25.997254 3934 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:26.000420 3952 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7e213916ed404f3e932b6fb7c9ed2c65. Latest consensus state: current_term: 1 leader_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } } }
I20260810 11:43:26.000648 3951 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7e213916ed404f3e932b6fb7c9ed2c65" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 44505 } } }
I20260810 11:43:26.001220 3952 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:26.001220 3951 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:26.004213 3964 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:26.009418 3964 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:26.019393 3964 catalog_manager.cc:1347] Generated new cluster ID: 0634612dd3d5413c894b872f6297fb69
I20260810 11:43:26.019577 3964 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:26.048388 3964 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:26.048864 3964 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:26.059587 3964 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7e213916ed404f3e932b6fb7c9ed2c65: Generated new TSK 0
I20260810 11:43:26.060431 3964 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:26.068758 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3825
W20260810 11:43:26.069864 3928 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38101
W20260810 11:43:26.070549 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 3825: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:26.070703 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 3825
2026-08-10T11:43:26Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2363 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:26 dist-test-slave-drl0 krb5kdc[3981](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:26 dist-test-slave-drl0 krb5kdc[3981](info): set up 2 sockets
Aug 10 11:43:26 dist-test-slave-drl0 krb5kdc[3981](info): commencing operation
krb5kdc: starting...
W20260810 11:43:27.457765 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.339s user 0.000s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:27 dist-test-slave-drl0 krb5kdc[3981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:27Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:27.835333 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40069
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:41833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:40069
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:28.220402 4003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:28.220778 4003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:28.220980 4003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:28.229707 4003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:28.229892 4003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:28.230011 4003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:28.230118 4003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:28.230208 4003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:28.241289 4003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:41833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:40069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:40069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:28.243011 4003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:28.244432 4003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:28.257827 4011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:28.257859 4015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:28.257786 4010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:28.259287 4003 server_base.cc:1034] running on GCE node
I20260810 11:43:28.260102 4003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:28.261862 4003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:28.263093 4003 hybrid_clock.cc:648] HybridClock initialized: now 1786362208263051 us; error 57 us; skew 500 ppm
I20260810 11:43:28.263480 4003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:28 dist-test-slave-drl0 krb5kdc[3981](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362208, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:28.268059 4003 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:28.270223 4003 webserver.cc:460] Webserver started at http://127.3.240.254:38989/ using document root <none> and password file <none>
I20260810 11:43:28.271251 4003 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:28.271435 4003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:28.271773 4003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:28.274890 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e"
format_stamp: "Formatted at 2026-08-10 11:43:28 on dist-test-slave-drl0"
server_key: "b58e7b30e4f8a590f0502152bec5d17b"
server_key_iv: "4e95febbffa9e5542f04efcbb5e27f17"
server_key_version: "encryptionkey@0"
I20260810 11:43:28.275770 4003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e"
format_stamp: "Formatted at 2026-08-10 11:43:28 on dist-test-slave-drl0"
server_key: "b58e7b30e4f8a590f0502152bec5d17b"
server_key_iv: "4e95febbffa9e5542f04efcbb5e27f17"
server_key_version: "encryptionkey@0"
I20260810 11:43:28.282135 4003 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260810 11:43:28.286883 4025 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:28.288632 4003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:43:28.288887 4003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e"
format_stamp: "Formatted at 2026-08-10 11:43:28 on dist-test-slave-drl0"
server_key: "b58e7b30e4f8a590f0502152bec5d17b"
server_key_iv: "4e95febbffa9e5542f04efcbb5e27f17"
server_key_version: "encryptionkey@0"
I20260810 11:43:28.289121 4003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:28.308320 4003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:28.313433 4003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:28.313784 4003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:28.338296 4003 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:40069
I20260810 11:43:28.338400 4116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:40069 every 8 connection(s)
I20260810 11:43:28.339890 4003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:28.343643 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4003
I20260810 11:43:28.343838 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:28.344403 4035 external_mini_cluster.cc:1428] Setting key 9fa4511aced28fbada7a0b7894effb51
I20260810 11:43:28.344856 4117 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:28.358690 4117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e: Bootstrap starting.
Aug 10 11:43:28 dist-test-slave-drl0 krb5kdc[3981](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362207, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:28.362788 4117 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:28.364063 4117 log.cc:824] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:28.368399 4117 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e: No bootstrap required, opened a new log
I20260810 11:43:28.369706 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:28.346432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56483 (local address 127.3.240.254:40069)
0810 11:43:28.347342 (+ 910us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:28.347353 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:28.347398 (+ 45us) server_negotiation.cc:407] Connection header received
0810 11:43:28.349742 (+ 2344us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:28.349766 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:28.350199 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:28.350642 (+ 443us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:28.352258 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:28.353597 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:28.354956 (+ 1359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:28.355697 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:28.355934 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:28.360757 (+ 4823us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:28.360766 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:28.360779 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:28.360829 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:28.364916 (+ 4087us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:28.365960 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:28.365968 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:28.365974 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:28.366165 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:28.366605 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:28.366611 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:28.366615 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:28.367345 (+ 730us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:28.367678 (+ 333us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:28.368176 (+ 498us) server_negotiation.cc:299] Negotiation successful
0810 11:43:28.368666 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":690,"thread_start_us":438,"threads_started":1}
I20260810 11:43:28.374863 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:28.377354 4117 raft_consensus.cc:348] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } }
I20260810 11:43:28.377802 4117 raft_consensus.cc:374] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:28.377925 4117 raft_consensus.cc:729] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a0080c0f9f7d4b0ea39d810c1d658b6e, State: Initialized, Role: FOLLOWER
I20260810 11:43:28.378587 4117 consensus_queue.cc:260] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } }
I20260810 11:43:28.378829 4117 raft_consensus.cc:388] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:28.379009 4117 raft_consensus.cc:482] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:28.379189 4117 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:28.382710 4117 raft_consensus.cc:504] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } }
I20260810 11:43:28.383318 4117 leader_election.cc:302] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a0080c0f9f7d4b0ea39d810c1d658b6e; no voters:
I20260810 11:43:28.383968 4117 leader_election.cc:288] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:28.384124 4131 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:28.385792 4131 raft_consensus.cc:686] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [term 1 LEADER]: Becoming Leader. State: Replica: a0080c0f9f7d4b0ea39d810c1d658b6e, State: Running, Role: LEADER
I20260810 11:43:28.386548 4117 sys_catalog.cc:564] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:28.386500 4131 consensus_queue.cc:237] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } }
I20260810 11:43:28.392048 4132 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } } }
I20260810 11:43:28.392542 4132 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:28.403587 4122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:28.379713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52461 (local address 127.3.240.254:40069)
0810 11:43:28.379946 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:28.379955 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:28.380494 (+ 539us) server_negotiation.cc:407] Connection header received
0810 11:43:28.380669 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:28.380674 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:28.380745 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:28.380835 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:28.382828 (+ 1993us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:28.383908 (+ 1080us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:28.385784 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:28.386594 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:28.387121 (+ 527us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:28.390336 (+ 3215us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:28.390348 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:28.390356 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:28.390418 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:28.396750 (+ 6332us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:28.398140 (+ 1390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:28.398149 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:28.398155 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:28.398337 (+ 182us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:28.398962 (+ 625us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:28.398969 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:28.398975 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:28.399501 (+ 526us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:28.402023 (+ 2522us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:28.402650 (+ 627us) server_negotiation.cc:299] Negotiation successful
0810 11:43:28.403047 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:43:28.407335 4133 sys_catalog.cc:455] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [sys.catalog]: SysCatalogTable state changed. Reason: New leader a0080c0f9f7d4b0ea39d810c1d658b6e. Latest consensus state: current_term: 1 leader_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a0080c0f9f7d4b0ea39d810c1d658b6e" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 40069 } } }
I20260810 11:43:28.407792 4133 sys_catalog.cc:458] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:28.408219 4140 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:28.413959 4140 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:28.423722 4140 catalog_manager.cc:1347] Generated new cluster ID: 4a73deba08e04ff5a7fa747ab004d4a4
I20260810 11:43:28.423862 4140 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:28.449867 4140 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:28.451568 4140 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:28.459309 4140 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a0080c0f9f7d4b0ea39d810c1d658b6e: Generated new TSK 0
I20260810 11:43:28.460362 4140 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:28.474489 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4003
I20260810 11:43:28.475840 4109 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:28.575840 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4003
2026-08-10T11:43:28Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2504 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:28 dist-test-slave-drl0 krb5kdc[4159](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:28 dist-test-slave-drl0 krb5kdc[4159](info): set up 2 sockets
Aug 10 11:43:28 dist-test-slave-drl0 krb5kdc[4159](info): commencing operation
krb5kdc: starting...
W20260810 11:43:29.982911 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.005s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:30 dist-test-slave-drl0 krb5kdc[4159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362210, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:30Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:30.363333 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33929
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42553
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:33929
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:30.752432 4177 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:30.752729 4177 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:30.752912 4177 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:30.762051 4177 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:30.762195 4177 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:30.762305 4177 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:30.762413 4177 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:30.762494 4177 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:30.775345 4177 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42553
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:33929
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:33929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:30.776813 4177 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:30.778173 4177 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:30.792265 4183 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:30.792209 4188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:30.792290 4185 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:30.793074 4177 server_base.cc:1034] running on GCE node
I20260810 11:43:30.793889 4177 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:30.795467 4177 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:30.796803 4177 hybrid_clock.cc:648] HybridClock initialized: now 1786362210796728 us; error 62 us; skew 500 ppm
I20260810 11:43:30.797276 4177 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:30 dist-test-slave-drl0 krb5kdc[4159](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362210, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:30.805982 4177 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:30.807926 4177 webserver.cc:460] Webserver started at http://127.3.240.254:35635/ using document root <none> and password file <none>
I20260810 11:43:30.808681 4177 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:30.808857 4177 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:30.809224 4177 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:30.812670 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "6c45ec0ee07d4ed78667f1bc81daea44"
format_stamp: "Formatted at 2026-08-10 11:43:30 on dist-test-slave-drl0"
server_key: "9fc0c1673d595fee80091bdc0c989539"
server_key_iv: "8d11e9867940622ad8df092796f0750f"
server_key_version: "encryptionkey@0"
I20260810 11:43:30.813491 4177 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "6c45ec0ee07d4ed78667f1bc81daea44"
format_stamp: "Formatted at 2026-08-10 11:43:30 on dist-test-slave-drl0"
server_key: "9fc0c1673d595fee80091bdc0c989539"
server_key_iv: "8d11e9867940622ad8df092796f0750f"
server_key_version: "encryptionkey@0"
I20260810 11:43:30.819450 4177 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.004s
I20260810 11:43:30.824642 4201 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:30.826359 4177 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:43:30.826578 4177 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "6c45ec0ee07d4ed78667f1bc81daea44"
format_stamp: "Formatted at 2026-08-10 11:43:30 on dist-test-slave-drl0"
server_key: "9fc0c1673d595fee80091bdc0c989539"
server_key_iv: "8d11e9867940622ad8df092796f0750f"
server_key_version: "encryptionkey@0"
I20260810 11:43:30.826817 4177 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:30.844216 4177 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:30.848937 4177 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:30.849285 4177 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:30.872591 4177 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:33929
I20260810 11:43:30.872614 4288 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:33929 every 8 connection(s)
I20260810 11:43:30.874548 4177 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:30.878674 4289 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:30.882552 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4177
I20260810 11:43:30.882735 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:30.883435 4035 external_mini_cluster.cc:1428] Setting key b5eaeb4d177375c4aa2331f626b2bf13
I20260810 11:43:30.893417 4289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44: Bootstrap starting.
I20260810 11:43:30.897030 4289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:30.898273 4289 log.cc:824] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:30.899420 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:30.885578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38953 (local address 127.3.240.254:33929)
0810 11:43:30.886646 (+ 1068us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:30.886657 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:30.886704 (+ 47us) server_negotiation.cc:407] Connection header received
0810 11:43:30.888613 (+ 1909us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:30.888638 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:30.889040 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:30.889383 (+ 343us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:30.890027 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:30.891510 (+ 1483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:30.893489 (+ 1979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:30.894393 (+ 904us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:30.894542 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:30.896142 (+ 1600us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:30.896161 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:30.896176 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:30.896720 (+ 544us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:30.896802 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:30.896831 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:30.896933 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:30.897379 (+ 446us) server_negotiation.cc:299] Negotiation successful
0810 11:43:30.897756 (+ 377us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":591,"threads_started":1}
I20260810 11:43:30.902292 4289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44: No bootstrap required, opened a new log
I20260810 11:43:30.904124 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:30.912213 4289 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260810 11:43:30.912652 4289 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:30.912775 4289 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c45ec0ee07d4ed78667f1bc81daea44, State: Initialized, Role: FOLLOWER
I20260810 11:43:30.913416 4289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260810 11:43:30.913688 4289 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:30.913842 4289 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:30.914074 4289 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:30.915876 4297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:30.909143 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50695 (local address 127.3.240.254:33929)
0810 11:43:30.909357 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:30.909365 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:30.909744 (+ 379us) server_negotiation.cc:407] Connection header received
0810 11:43:30.909976 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:30.909980 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:30.910047 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:30.910166 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:30.910808 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:30.911609 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:30.913162 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:30.913695 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:30.913851 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:30.915135 (+ 1284us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:30.915143 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:30.915146 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:30.915167 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:30.915204 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:30.915225 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:30.915341 (+ 116us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:30.915472 (+ 131us) server_negotiation.cc:299] Negotiation successful
0810 11:43:30.915577 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260810 11:43:30.917023 4289 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260810 11:43:30.917611 4289 leader_election.cc:302] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6c45ec0ee07d4ed78667f1bc81daea44; no voters:
I20260810 11:43:30.918242 4289 leader_election.cc:288] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:30.918586 4305 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:30.919699 4305 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [term 1 LEADER]: Becoming Leader. State: Replica: 6c45ec0ee07d4ed78667f1bc81daea44, State: Running, Role: LEADER
I20260810 11:43:30.920317 4305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } }
I20260810 11:43:30.920562 4289 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:30.924319 4308 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6c45ec0ee07d4ed78667f1bc81daea44. Latest consensus state: current_term: 1 leader_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } } }
I20260810 11:43:30.924595 4306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6c45ec0ee07d4ed78667f1bc81daea44" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 33929 } } }
I20260810 11:43:30.924897 4308 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:30.924942 4306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:30.927863 4314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:30.938031 4314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:30.950660 4314 catalog_manager.cc:1347] Generated new cluster ID: 87bb40c6120b468c9a29b6e2a03ed23a
I20260810 11:43:30.950913 4314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:30.972170 4314 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:30.973666 4314 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:30.981928 4314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6c45ec0ee07d4ed78667f1bc81daea44: Generated new TSK 0
I20260810 11:43:30.983206 4314 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:30.996696 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4177
W20260810 11:43:30.998536 4283 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38953
W20260810 11:43:30.999768 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4177: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:31.000029 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4177
2026-08-10T11:43:31Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2427 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:31 dist-test-slave-drl0 krb5kdc[4334](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:31 dist-test-slave-drl0 krb5kdc[4334](info): set up 2 sockets
Aug 10 11:43:31 dist-test-slave-drl0 krb5kdc[4334](info): commencing operation
krb5kdc: starting...
W20260810 11:43:32.418033 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.367s user 0.004s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:32 dist-test-slave-drl0 krb5kdc[4334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362212, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:32.795482 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39251
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33345
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39251
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:33.186303 4354 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:33.186686 4354 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:33.186877 4354 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:33.194711 4354 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:33.194891 4354 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:33.195024 4354 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:33.195140 4354 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:33.195230 4354 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:33.205844 4354 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33345
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39251
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39251
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:33.207347 4354 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:33.208707 4354 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:33.221990 4362 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:33.222028 4360 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:33.222218 4359 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:33.223266 4354 server_base.cc:1034] running on GCE node
I20260810 11:43:33.224051 4354 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:33.225497 4354 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:33.226822 4354 hybrid_clock.cc:648] HybridClock initialized: now 1786362213226702 us; error 72 us; skew 500 ppm
I20260810 11:43:33.227512 4354 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:33 dist-test-slave-drl0 krb5kdc[4334](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362213, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:33.231472 4354 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:33.233093 4354 webserver.cc:460] Webserver started at http://127.3.240.254:43851/ using document root <none> and password file <none>
I20260810 11:43:33.233841 4354 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:33.233960 4354 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:33.234318 4354 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:33.237483 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "793227077ef547d9b889cf91028757f8"
format_stamp: "Formatted at 2026-08-10 11:43:33 on dist-test-slave-drl0"
server_key: "223e8e311d88c80f135dfe07478100bc"
server_key_iv: "c0ee2dbe1407dfd4bd6c14643c3047f5"
server_key_version: "encryptionkey@0"
I20260810 11:43:33.238269 4354 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "793227077ef547d9b889cf91028757f8"
format_stamp: "Formatted at 2026-08-10 11:43:33 on dist-test-slave-drl0"
server_key: "223e8e311d88c80f135dfe07478100bc"
server_key_iv: "c0ee2dbe1407dfd4bd6c14643c3047f5"
server_key_version: "encryptionkey@0"
I20260810 11:43:33.244163 4354 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:43:33.249137 4372 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:33.250753 4354 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:43:33.251073 4354 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "793227077ef547d9b889cf91028757f8"
format_stamp: "Formatted at 2026-08-10 11:43:33 on dist-test-slave-drl0"
server_key: "223e8e311d88c80f135dfe07478100bc"
server_key_iv: "c0ee2dbe1407dfd4bd6c14643c3047f5"
server_key_version: "encryptionkey@0"
I20260810 11:43:33.251302 4354 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:33.257541 4354 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:33.262616 4354 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:33.263285 4354 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:33.285538 4354 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39251
I20260810 11:43:33.285581 4452 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39251 every 8 connection(s)
I20260810 11:43:33.287056 4354 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:33.291824 4453 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:33.295006 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4354
I20260810 11:43:33.295234 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:33.295693 4035 external_mini_cluster.cc:1428] Setting key 0814a41b37a2e2253977d42d6dab2a96
I20260810 11:43:33.305187 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8: Bootstrap starting.
I20260810 11:43:33.305572 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:33.297612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57103 (local address 127.3.240.254:39251)
0810 11:43:33.298510 (+ 898us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:33.298523 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:33.298571 (+ 48us) server_negotiation.cc:407] Connection header received
0810 11:43:33.300817 (+ 2246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:33.300852 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:33.301368 (+ 516us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:33.301961 (+ 593us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:33.303353 (+ 1392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:33.303359 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:33.303374 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:33.303393 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:33.303455 (+ 62us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:33.303489 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:33.303839 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:33.304014 (+ 175us) server_negotiation.cc:299] Negotiation successful
0810 11:43:33.304216 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":679,"thread_start_us":429,"threads_started":1}
I20260810 11:43:33.308856 4453 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:33.309954 4453 log.cc:824] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:33.310307 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:33.312934 4453 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8: No bootstrap required, opened a new log
I20260810 11:43:33.317858 4459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:33.315049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50331 (local address 127.3.240.254:39251)
0810 11:43:33.315280 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:33.315287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:33.315696 (+ 409us) server_negotiation.cc:407] Connection header received
0810 11:43:33.315835 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:33.315839 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:33.315889 (+ 50us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:33.315978 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:33.317009 (+ 1031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:33.317013 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:33.317017 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0810 11:43:33.317035 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:33.317077 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0810 11:43:33.317093 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:33.317197 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:33.317384 (+ 187us) server_negotiation.cc:299] Negotiation successful
0810 11:43:33.317431 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260810 11:43:33.322018 4453 raft_consensus.cc:348] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } }
I20260810 11:43:33.322400 4453 raft_consensus.cc:374] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260810 11:43:33.322505 4453 raft_consensus.cc:729] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 793227077ef547d9b889cf91028757f8, State: Initialized, Role: FOLLOWER
I20260810 11:43:33.323170 4453 consensus_queue.cc:260] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } }
I20260810 11:43:33.323541 4453 raft_consensus.cc:388] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260810 11:43:33.323702 4453 raft_consensus.cc:482] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260810 11:43:33.323917 4453 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:33.326097 4453 raft_consensus.cc:504] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } }
I20260810 11:43:33.326629 4453 leader_election.cc:302] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 793227077ef547d9b889cf91028757f8; no voters:
I20260810 11:43:33.327713 4453 leader_election.cc:288] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260810 11:43:33.327900 4468 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:33.329123 4468 raft_consensus.cc:686] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [term 1 LEADER]: Becoming Leader. State: Replica: 793227077ef547d9b889cf91028757f8, State: Running, Role: LEADER
I20260810 11:43:33.329852 4468 consensus_queue.cc:237] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } }
I20260810 11:43:33.330873 4453 sys_catalog.cc:564] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:33.333504 4470 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "793227077ef547d9b889cf91028757f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } } }
I20260810 11:43:33.334100 4470 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:33.333917 4472 sys_catalog.cc:455] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 793227077ef547d9b889cf91028757f8. Latest consensus state: current_term: 1 leader_uuid: "793227077ef547d9b889cf91028757f8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "793227077ef547d9b889cf91028757f8" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39251 } } }
I20260810 11:43:33.334326 4472 sys_catalog.cc:458] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:33.340191 4479 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:33.347743 4479 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:33.358830 4479 catalog_manager.cc:1347] Generated new cluster ID: 2f8e6debef7f4ed192e4efbe85849dec
I20260810 11:43:33.359041 4479 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:33.388497 4479 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:33.389307 4479 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:33.413614 4479 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 793227077ef547d9b889cf91028757f8: Generated new TSK 0
I20260810 11:43:33.414525 4479 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260810 11:43:33.432005 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4354
W20260810 11:43:33.434185 4439 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:57103
W20260810 11:43:33.434913 4035 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4354: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260810 11:43:33.435099 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4354
2026-08-10T11:43:33Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2439 ms)
[----------] 18 tests from AuthTokenIssuingTest (43800 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:33 dist-test-slave-drl0 krb5kdc[4497](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:33 dist-test-slave-drl0 krb5kdc[4497](info): set up 2 sockets
Aug 10 11:43:33 dist-test-slave-drl0 krb5kdc[4497](info): commencing operation
krb5kdc: starting...
W20260810 11:43:34.822149 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.331s user 0.001s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:35 dist-test-slave-drl0 krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:35Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:35.207445 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45507
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:35.577039 4517 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:35.577435 4517 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:35.577569 4517 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:35.577732 4517 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:35.585443 4517 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:35.585539 4517 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:35.585600 4517 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:35.585736 4517 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:35.585834 4517 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:35.597564 4517 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:45507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:35.599032 4517 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:35.600406 4517 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:35.614836 4525 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:35.614885 4526 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:35.614758 4528 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:35.615849 4517 server_base.cc:1034] running on GCE node
I20260810 11:43:35.616678 4517 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:35.618100 4517 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:35.619392 4517 hybrid_clock.cc:648] HybridClock initialized: now 1786362215619302 us; error 55 us; skew 500 ppm
I20260810 11:43:35.619724 4517 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:35 dist-test-slave-drl0 krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:35.623227 4517 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:35.624867 4517 webserver.cc:460] Webserver started at http://127.3.240.254:38637/ using document root <none> and password file <none>
I20260810 11:43:35.625600 4517 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:35.625694 4517 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:35.626024 4517 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:35.629087 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "9e6786908867457a8ce9191ccbe08fbf"
format_stamp: "Formatted at 2026-08-10 11:43:35 on dist-test-slave-drl0"
server_key: "f2db7780fe2e5e986c1216d4f656f8e4"
server_key_iv: "4b9bd7dbe6ab719312cc81ca53d5a15b"
server_key_version: "encryptionkey@0"
I20260810 11:43:35.629899 4517 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "9e6786908867457a8ce9191ccbe08fbf"
format_stamp: "Formatted at 2026-08-10 11:43:35 on dist-test-slave-drl0"
server_key: "f2db7780fe2e5e986c1216d4f656f8e4"
server_key_iv: "4b9bd7dbe6ab719312cc81ca53d5a15b"
server_key_version: "encryptionkey@0"
I20260810 11:43:35.635720 4517 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260810 11:43:35.640035 4538 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:35.641855 4517 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:43:35.642076 4517 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "9e6786908867457a8ce9191ccbe08fbf"
format_stamp: "Formatted at 2026-08-10 11:43:35 on dist-test-slave-drl0"
server_key: "f2db7780fe2e5e986c1216d4f656f8e4"
server_key_iv: "4b9bd7dbe6ab719312cc81ca53d5a15b"
server_key_version: "encryptionkey@0"
I20260810 11:43:35.642309 4517 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:35.659263 4517 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:35.664074 4517 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:35.664361 4517 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:35.685694 4517 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:45507
I20260810 11:43:35.685722 4611 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:45507 every 8 connection(s)
I20260810 11:43:35.687250 4517 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:35.688571 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4517
I20260810 11:43:35.688820 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:35.689875 4035 external_mini_cluster.cc:1428] Setting key d8f15daad40474b246383cfedc7cd2ce
I20260810 11:43:35.692548 4612 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:35.703656 4612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:35 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
I20260810 11:43:35.732744 4616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.708188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39370 (local address 127.3.240.254:45507)
0810 11:43:35.709391 (+ 1203us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:35.709396 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:35.709447 (+ 51us) server_negotiation.cc:407] Connection header received
0810 11:43:35.712128 (+ 2681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:35.712134 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:35.712352 (+ 218us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:35.712515 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:35.714307 (+ 1792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:35.715679 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:35.718833 (+ 3154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:35.719499 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:35.719635 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:35.724552 (+ 4917us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:35.724560 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:35.724574 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:35.724606 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:35.727876 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:35.728647 (+ 771us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:35.728664 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:35.728667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:35.728835 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:35.729289 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:35.729296 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:35.729305 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:35.729865 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:35.730091 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:35.730898 (+ 807us) server_negotiation.cc:299] Negotiation successful
0810 11:43:35.731249 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1041,"thread_start_us":897,"threads_started":1}
I20260810 11:43:35.732798 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.707930 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45507 (local address 127.0.0.1:39370)
0810 11:43:35.708839 (+ 909us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.708885 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:35.711714 (+ 2829us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:35.712607 (+ 893us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:35.712617 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:35.713454 (+ 837us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:35.714042 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:35.714075 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:35.716019 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:35.716031 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:35.718605 (+ 2574us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:35.718631 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:35.719788 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:35.719799 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:35.720129 (+ 330us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:35.720954 (+ 825us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:35.720989 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:35.724272 (+ 3283us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:35.728026 (+ 3754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:35.728041 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:35.728054 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:35.728497 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:35.728969 (+ 472us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:35.728974 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:35.728978 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:35.729152 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:35.730043 (+ 891us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:35.730052 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:35.730285 (+ 233us) client_negotiation.cc:769] Sending connection context
0810 11:43:35.730614 (+ 329us) client_negotiation.cc:240] Negotiation successful
0810 11:43:35.731379 (+ 765us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":634,"mutex_wait_us":161,"thread_start_us":403,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
I20260810 11:43:35.736063 4612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:35.737202 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.736780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53608)
0810 11:43:35.736976 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.737088 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20260810 11:43:35.737453 4541 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
I20260810 11:43:35.739092 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.738680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53616)
0810 11:43:35.738842 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.738905 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 11:43:35.739322 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:45917: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260810 11:43:35.779392 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:45917
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:43:35.788656 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } attempt: 1
I20260810 11:43:35.790019 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.789400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53622)
0810 11:43:35.789658 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.789809 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 11:43:35.791375 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.790856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53624)
0810 11:43:35.791066 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.791162 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260810 11:43:35.791662 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:45917: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
I20260810 11:43:35.859970 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } attempt: 2
I20260810 11:43:35.861392 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.860755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53630)
0810 11:43:35.861019 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.861154 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260810 11:43:35.862900 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.862413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53646)
0810 11:43:35.862588 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.862705 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260810 11:43:35.863161 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:45917: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
I20260810 11:43:35.954463 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } attempt: 3
I20260810 11:43:35.955727 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.955224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53662)
0810 11:43:35.955449 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.955560 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260810 11:43:35.956861 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:35.956430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53678)
0810 11:43:35.956592 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:35.956684 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 11:43:35.957139 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:45917: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
I20260810 11:43:36.100445 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } attempt: 4
I20260810 11:43:36.102046 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.101402 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53688)
0810 11:43:36.101740 (+ 338us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.101860 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":163}
I20260810 11:43:36.103572 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.103096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53698)
0810 11:43:36.103286 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.103396 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 11:43:36.104054 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:45917: Network error: Client connection negotiation failed: client connection to 127.3.240.253:45917: connect: Connection refused (error 111)
W20260810 11:43:36.156380 4624 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:36.156708 4624 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:36.156778 4624 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:36.156862 4624 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:36.165292 4624 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:36.165437 4624 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:36.165540 4624 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:36.165598 4624 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:36.165656 4624 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:36.176623 4624 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:45917
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:36.178041 4624 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:36.179420 4624 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:36.193835 4635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:36.193905 4634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:36.193873 4637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:36.195243 4624 server_base.cc:1034] running on GCE node
I20260810 11:43:36.196110 4624 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:36.197523 4624 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:36.198884 4624 hybrid_clock.cc:648] HybridClock initialized: now 1786362216198791 us; error 59 us; skew 500 ppm
I20260810 11:43:36.199649 4624 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:36.206360 4624 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260810 11:43:36.208346 4624 webserver.cc:460] Webserver started at http://127.3.240.253:33613/ using document root <none> and password file <none>
I20260810 11:43:36.209081 4624 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:36.209224 4624 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:36.209573 4624 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:36.213223 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data/instance:
uuid: "8766dd91e26f49558745c4772318b2e9"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "cf5db1de6ee81a034512d93a7498633a"
server_key_iv: "691b6ccb24a2ca09d92c4d5e97db60aa"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.214551 4624 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal/instance:
uuid: "8766dd91e26f49558745c4772318b2e9"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "cf5db1de6ee81a034512d93a7498633a"
server_key_iv: "691b6ccb24a2ca09d92c4d5e97db60aa"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.220235 4624 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260810 11:43:36.224301 4644 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:36.226173 4624 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260810 11:43:36.226495 4624 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "8766dd91e26f49558745c4772318b2e9"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "cf5db1de6ee81a034512d93a7498633a"
server_key_iv: "691b6ccb24a2ca09d92c4d5e97db60aa"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.226820 4624 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:36.259227 4624 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:36.264557 4624 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:36.264858 4624 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:36.287365 4624 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:45917
I20260810 11:43:36.287426 4722 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:45917 every 8 connection(s)
I20260810 11:43:36.288904 4624 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
I20260810 11:43:36.292884 4723 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:36.298095 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4624
I20260810 11:43:36.298326 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-1/wal/instance
I20260810 11:43:36.298882 4035 external_mini_cluster.cc:1428] Setting key e5779bf444c230296f38f3105eb24910
I20260810 11:43:36.300354 4723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:36.322680 4725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.303049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39376 (local address 127.3.240.254:45507)
0810 11:43:36.303627 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.303634 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.304558 (+ 924us) server_negotiation.cc:407] Connection header received
0810 11:43:36.306409 (+ 1851us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.306417 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.306512 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.306648 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.308256 (+ 1608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.309509 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.311135 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.312276 (+ 1141us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.312630 (+ 354us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.316354 (+ 3724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.316358 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.316362 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.316399 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.319370 (+ 2971us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.320140 (+ 770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.320148 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.320152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.320289 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.320824 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.320830 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.320834 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.321158 (+ 324us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.321345 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.322060 (+ 715us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.322282 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":280,"threads_started":1}
I20260810 11:43:36.323740 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.303571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45507 (local address 127.0.0.1:39376)
0810 11:43:36.304376 (+ 805us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.304431 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.306138 (+ 1707us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.306852 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.306860 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.307469 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.308037 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.308055 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.309712 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.309716 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.310916 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.310991 (+ 75us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.312728 (+ 1737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.312733 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.312941 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.313656 (+ 715us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.313679 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.316084 (+ 2405us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.319520 (+ 3436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.319529 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.319546 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.320002 (+ 456us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.320409 (+ 407us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.320415 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.320418 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.320598 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.321308 (+ 710us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.321317 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.321687 (+ 370us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.321978 (+ 291us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.322484 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":550,"thread_start_us":333,"threads_started":1}
I20260810 11:43:36.325706 4723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:36.351958 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.326473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53714)
0810 11:43:36.326672 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.326699 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.326788 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.328003 (+ 1215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.328008 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.328552 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.328694 (+ 142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.328703 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.330083 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.330091 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.331468 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.331499 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.332795 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.332801 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.333017 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.333994 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.334016 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.341004 (+ 6988us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.348665 (+ 7661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.348669 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.348672 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.349104 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.349695 (+ 591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.349701 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.349704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.350008 (+ 304us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.351172 (+ 1164us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.351177 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.351310 (+ 133us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.351444 (+ 134us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.351675 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80}
I20260810 11:43:36.352430 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.326859 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53714 (local address 127.3.240.253:45917)
0810 11:43:36.327570 (+ 711us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.327576 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.327601 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:43:36.327643 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.327648 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.327837 (+ 189us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.328107 (+ 270us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.328885 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.329946 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.331747 (+ 1801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.332602 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.332765 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.341797 (+ 9032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.341820 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.341844 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.341903 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.348461 (+ 6558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.349253 (+ 792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.349266 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.349272 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.349426 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.350251 (+ 825us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.350259 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.350265 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.350824 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.351152 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.351645 (+ 493us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.351956 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":565,"thread_start_us":423,"threads_started":1}
I20260810 11:43:36.355469 4723 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:36.356758 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.356259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41372)
0810 11:43:36.356467 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.356609 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260810 11:43:36.357059 4646 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.358587 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.358264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41378)
0810 11:43:36.358417 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.358490 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 11:43:36.358798 4723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
I20260810 11:43:36.387691 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:37507
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:33271
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:43:36.403398 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } attempt: 5
I20260810 11:43:36.408156 4723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 1
I20260810 11:43:36.409332 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.408758 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41382)
0810 11:43:36.408995 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.409139 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260810 11:43:36.410645 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.410195 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41384)
0810 11:43:36.410395 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.410494 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 11:43:36.411109 4723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:36.420614 4728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.404569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53724 (local address 127.3.240.253:45917)
0810 11:43:36.404814 (+ 245us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.404823 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.404853 (+ 30us) server_negotiation.cc:407] Connection header received
0810 11:43:36.405016 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.405024 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.405144 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.405294 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.406342 (+ 1048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.408202 (+ 1860us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.409901 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.410558 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.410721 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.414780 (+ 4059us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.414787 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.414790 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.414834 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.417753 (+ 2919us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.418551 (+ 798us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.418558 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.418562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.418690 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.419182 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.419188 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.419192 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.419481 (+ 289us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.419646 (+ 165us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.420047 (+ 401us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.420295 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260810 11:43:36.420632 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.404449 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53724)
0810 11:43:36.404694 (+ 245us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.404721 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.404865 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.405403 (+ 538us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.405409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.405810 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.406138 (+ 328us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.406155 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.408382 (+ 2227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.408387 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.409711 (+ 1324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.409726 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.410752 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.410759 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.411090 (+ 331us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.411765 (+ 675us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.411784 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.414483 (+ 2699us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.417939 (+ 3456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.417945 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.417949 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.418392 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.418835 (+ 443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.418840 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.418844 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.419069 (+ 225us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.419613 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.419618 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.419790 (+ 172us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.419942 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.420271 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20260810 11:43:36.421927 4612 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:36.423475 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.422888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41394)
0810 11:43:36.423188 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.423294 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260810 11:43:36.424865 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.424424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41396)
0810 11:43:36.424607 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.424723 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260810 11:43:36.425155 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.476433 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 1
I20260810 11:43:36.478046 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.477423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41400)
0810 11:43:36.477691 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.477813 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260810 11:43:36.479372 4723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 2
I20260810 11:43:36.479811 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.479162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41410)
0810 11:43:36.479498 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.479624 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
W20260810 11:43:36.480182 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.480927 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.480486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41422)
0810 11:43:36.480694 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.480775 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260810 11:43:36.482023 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.481671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41430)
0810 11:43:36.481850 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.481918 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
W20260810 11:43:36.482281 4723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.548483 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 2
I20260810 11:43:36.550186 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.549304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41446)
0810 11:43:36.549792 (+ 488us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.549934 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260810 11:43:36.551759 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.551331 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41452)
0810 11:43:36.551515 (+ 184us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.551611 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260810 11:43:36.552156 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.573547 4723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 3
I20260810 11:43:36.574852 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.574270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41468)
0810 11:43:36.574498 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.574583 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20260810 11:43:36.576589 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.575856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41476)
0810 11:43:36.576391 (+ 535us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.576474 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417}
W20260810 11:43:36.576884 4723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.658468 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 3
I20260810 11:43:36.660074 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.659300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41478)
0810 11:43:36.659591 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.659719 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
I20260810 11:43:36.661370 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.660800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41490)
0810 11:43:36.661093 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.661213 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
W20260810 11:43:36.661737 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.720175 4723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 4
I20260810 11:43:36.721328 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.720838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41498)
0810 11:43:36.721088 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.721170 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20260810 11:43:36.722249 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.721947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41508)
0810 11:43:36.722086 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.722155 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 11:43:36.722783 4723 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
W20260810 11:43:36.799032 4731 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:36.799358 4731 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:36.799476 4731 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:36.799628 4731 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:36.809422 4731 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:36.809535 4731 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:36.809605 4731 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:36.809706 4731 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:36.809830 4731 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:36.821942 4731 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:33271
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:45507,127.3.240.253:45917,127.3.240.252:37507
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:37507
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:36.823354 4731 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:36.824699 4731 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:36.837309 4736 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:36.837355 4737 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:36.837391 4739 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:36.838634 4731 server_base.cc:1034] running on GCE node
I20260810 11:43:36.839035 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 4
I20260810 11:43:36.839931 4731 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:36.840416 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.839658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41516)
0810 11:43:36.840103 (+ 445us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.840229 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":289}
I20260810 11:43:36.841565 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.841140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41530)
0810 11:43:36.841316 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.841402 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 11:43:36.841841 4731 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260810 11:43:36.841878 4612 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:37507: Network error: Client connection negotiation failed: client connection to 127.3.240.252:37507: connect: Connection refused (error 111)
I20260810 11:43:36.843078 4731 hybrid_clock.cc:648] HybridClock initialized: now 1786362216842990 us; error 50 us; skew 500 ppm
I20260810 11:43:36.843467 4731 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:36.847203 4731 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260810 11:43:36.849116 4731 webserver.cc:460] Webserver started at http://127.3.240.252:36425/ using document root <none> and password file <none>
I20260810 11:43:36.850303 4731 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:36.850528 4731 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:36.851097 4731 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:36.854187 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data/instance:
uuid: "19727e15def443b69d7c787b86ff9ed1"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "292713a8132359c2c06720fb2fcd361a"
server_key_iv: "97075aeabb3e40956cea9f4632fea7ac"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.855126 4731 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal/instance:
uuid: "19727e15def443b69d7c787b86ff9ed1"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "292713a8132359c2c06720fb2fcd361a"
server_key_iv: "97075aeabb3e40956cea9f4632fea7ac"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.861123 4731 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260810 11:43:36.865351 4749 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:36.867210 4731 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:43:36.867403 4731 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "19727e15def443b69d7c787b86ff9ed1"
format_stamp: "Formatted at 2026-08-10 11:43:36 on dist-test-slave-drl0"
server_key: "292713a8132359c2c06720fb2fcd361a"
server_key_iv: "97075aeabb3e40956cea9f4632fea7ac"
server_key_version: "encryptionkey@0"
I20260810 11:43:36.867628 4731 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:36.888222 4731 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:36.894104 4731 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:36.894507 4731 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:36.916715 4731 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:37507
I20260810 11:43:36.916759 4826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:37507 every 8 connection(s)
I20260810 11:43:36.918788 4731 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
I20260810 11:43:36.922466 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 4731
I20260810 11:43:36.922663 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786361809851056-4035-0/minicluster-data/master-2/wal/instance
I20260810 11:43:36.922968 4829 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:36.923278 4035 external_mini_cluster.cc:1428] Setting key 030d3982390973e8ea4d0ad105e71c30
I20260810 11:43:36.934249 4829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:36.954324 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.925659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41544 (local address 127.3.240.252:37507)
0810 11:43:36.926554 (+ 895us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.926566 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.926630 (+ 64us) server_negotiation.cc:407] Connection header received
0810 11:43:36.930154 (+ 3524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.930185 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.930712 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.931242 (+ 530us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.933321 (+ 2079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.935555 (+ 2234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.937203 (+ 1648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.938293 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.938538 (+ 245us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.945458 (+ 6920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.945472 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.945491 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.945552 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.949642 (+ 4090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.950646 (+ 1004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.950653 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.950660 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.950823 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.951308 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.951312 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.951314 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.951874 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.952257 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.952832 (+ 575us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.953254 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":666,"thread_start_us":436,"threads_started":1}
I20260810 11:43:36.954466 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.937229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:45507 (local address 127.0.0.1:39384)
0810 11:43:36.938092 (+ 863us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.938143 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.938308 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.939387 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.939394 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.940300 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.940686 (+ 386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.940701 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.941777 (+ 1076us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.941783 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.942780 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.942792 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.943607 (+ 815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.943614 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.943777 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.944465 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.944487 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.947765 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.951682 (+ 3917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.951693 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.951701 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.952153 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.952554 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.952560 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.952568 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.952706 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.953331 (+ 625us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.953338 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.953503 (+ 165us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.953690 (+ 187us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.954033 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":591,"thread_start_us":400,"threads_started":1}
I20260810 11:43:36.954465 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.937272 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39384 (local address 127.3.240.254:45507)
0810 11:43:36.938975 (+ 1703us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.938985 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.939008 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:43:36.939075 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.939080 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.939178 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.939303 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.940860 (+ 1557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.941596 (+ 736us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.942968 (+ 1372us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.943376 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.943529 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.948022 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.948028 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.948031 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.948065 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.951548 (+ 3483us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.952281 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.952287 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.952291 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.952429 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.952836 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.952842 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.952845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.953188 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.953346 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.953798 (+ 452us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.954025 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1554,"thread_start_us":1319,"threads_started":1}
I20260810 11:43:36.957041 4829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:36.976275 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.957776 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:45917 (local address 127.0.0.1:53730)
0810 11:43:36.958234 (+ 458us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.958257 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.958358 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.958837 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.958841 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.959250 (+ 409us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.959381 (+ 131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.959397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.960426 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.960431 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.961889 (+ 1458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.961905 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.963614 (+ 1709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.963621 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.963812 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.965275 (+ 1463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.965312 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.968664 (+ 3352us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.973052 (+ 4388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.973056 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.973059 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.973628 (+ 569us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.974045 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.974050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.974055 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.974243 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.975033 (+ 790us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.975046 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.975326 (+ 280us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.975499 (+ 173us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.975791 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344}
I20260810 11:43:36.976776 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.958056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53730 (local address 127.3.240.253:45917)
0810 11:43:36.958569 (+ 513us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.958574 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.958589 (+ 15us) server_negotiation.cc:407] Connection header received
0810 11:43:36.958635 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.958639 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.958694 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.958813 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.959581 (+ 768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.960274 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.962082 (+ 1808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.963295 (+ 1213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.963453 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.968906 (+ 5453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.968911 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.968915 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.968949 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.972871 (+ 3922us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.973770 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.973774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.973777 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.973912 (+ 135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.974374 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.974380 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.974386 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.974737 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.974864 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.975888 (+ 1024us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.976272 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260810 11:43:36.977949 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.957776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39400 (local address 127.3.240.254:45507)
0810 11:43:36.957959 (+ 183us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.957965 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.957988 (+ 23us) server_negotiation.cc:407] Connection header received
0810 11:43:36.958149 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.958154 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.958259 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.959664 (+ 1405us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.959771 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.960446 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.962210 (+ 1764us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.963373 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.963502 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.969831 (+ 6329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.969837 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.969842 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.969882 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.973487 (+ 3605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.974296 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.974333 (+ 37us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.974336 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.974507 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.975349 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.975356 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.975358 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.976091 (+ 733us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.976464 (+ 373us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.977338 (+ 874us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.977604 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:43:36.978063 4829 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:43:36 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:36.994390 4837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.979153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41552)
0810 11:43:36.979442 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:36.979459 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:36.979604 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:36.979910 (+ 306us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:36.979914 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:36.980450 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:36.980615 (+ 165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.980625 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.981662 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.981669 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.982515 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:36.982525 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.983284 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.983290 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:36.983467 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.984020 (+ 553us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:36.984038 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:36.987192 (+ 3154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:36.991085 (+ 3893us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.991092 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.991096 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.991619 (+ 523us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.992225 (+ 606us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:36.992232 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:36.992238 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:36.992458 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:36.993161 (+ 703us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:36.993165 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:36.993382 (+ 217us) client_negotiation.cc:769] Sending connection context
0810 11:43:36.993594 (+ 212us) client_negotiation.cc:240] Negotiation successful
0810 11:43:36.993885 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":198}
I20260810 11:43:36.994570 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.979153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41552 (local address 127.3.240.252:37507)
0810 11:43:36.979366 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.979372 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.979529 (+ 157us) server_negotiation.cc:407] Connection header received
0810 11:43:36.979734 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.979738 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.979795 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.979893 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.980788 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.981522 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.982652 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.983128 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.983263 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.987429 (+ 4166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.987434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.987438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.987470 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.990840 (+ 3370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.991835 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.991841 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.991845 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.992007 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.992585 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.992591 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.992594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.992982 (+ 388us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.993163 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.993858 (+ 695us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.994177 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:43:36.998538 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:36.980304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53732 (local address 127.3.240.253:45917)
0810 11:43:36.980490 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:36.980495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:36.980509 (+ 14us) server_negotiation.cc:407] Connection header received
0810 11:43:36.980546 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:36.980549 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:36.980602 (+ 53us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:36.980709 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:36.982346 (+ 1637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.983157 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.984360 (+ 1203us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:36.984890 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:36.985456 (+ 566us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:36.991444 (+ 5988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:36.991450 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:36.991455 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:36.991517 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:36.995304 (+ 3787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.996238 (+ 934us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.996244 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.996248 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.996424 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:36.996877 (+ 453us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:36.996882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:36.996886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:36.997279 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:36.997577 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:36.997935 (+ 358us) server_negotiation.cc:299] Negotiation successful
0810 11:43:36.998182 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:43:37.001192 4829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: Bootstrap starting.
I20260810 11:43:37.004699 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:37.005106 4829 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:37.006160 4829 log.cc:824] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:37.009459 4829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: No bootstrap required, opened a new log
I20260810 11:43:37.008036 4576 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "9e6786908867457a8ce9191ccbe08fbf"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39400
I20260810 11:43:37.016956 4829 raft_consensus.cc:348] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.017320 4829 raft_consensus.cc:729] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 19727e15def443b69d7c787b86ff9ed1, State: Initialized, Role: FOLLOWER
I20260810 11:43:37.017752 4829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.018610 4843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.018977 4843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.019970 4829 sys_catalog.cc:564] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:37.022145 4723 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 5
W20260810 11:43:37.031051 4863 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:43:37.031234 4863 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
Aug 10 11:43:37 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362216, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260810 11:43:37.042219 4726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.022863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41564)
0810 11:43:37.023286 (+ 423us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:37.023307 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:37.023424 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:37.023736 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:37.023741 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:37.024379 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:37.024646 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:37.024659 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.026457 (+ 1798us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.026463 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:37.027461 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:37.027474 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.028736 (+ 1262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.028742 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:37.028983 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.029707 (+ 724us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:37.029732 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:37.032925 (+ 3193us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:37.037890 (+ 4965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:37.037896 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:37.037900 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:37.038481 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:37.038996 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:37.039000 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:37.039004 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:37.039191 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:37.040020 (+ 829us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:37.040028 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:37.040323 (+ 295us) client_negotiation.cc:769] Sending connection context
0810 11:43:37.041041 (+ 718us) client_negotiation.cc:240] Negotiation successful
0810 11:43:37.041776 (+ 735us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273}
I20260810 11:43:37.030061 4731 master_runner.cc:186] Error getting master registration for 127.3.240.254:45507: OK, instance_id { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" instance_seqno: 1786362215679085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:43:37.043200 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.023252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.3.240.252:37507)
0810 11:43:37.023441 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.023446 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.023468 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:43:37.023520 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.023524 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.023583 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.023678 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.024785 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.026067 (+ 1282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.027608 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.028337 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.028470 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.033157 (+ 4687us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:37.033170 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:37.033175 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:37.033219 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:37.037726 (+ 4507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.038628 (+ 902us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.038633 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.038637 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.038826 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.039337 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.039342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.039346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.039841 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:37.040002 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:37.041810 (+ 1808us) server_negotiation.cc:299] Negotiation successful
0810 11:43:37.042266 (+ 456us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20260810 11:43:37.043313 4731 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 11:43:37.048156 4723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: Bootstrap starting.
I20260810 11:43:37.052500 4723 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:37.053615 4723 log.cc:824] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:37.057056 4723 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: No bootstrap required, opened a new log
I20260810 11:43:37.064637 4723 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.065003 4723 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8766dd91e26f49558745c4772318b2e9, State: Initialized, Role: FOLLOWER
I20260810 11:43:37.065793 4723 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.067632 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.068171 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.069093 4723 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 11:43:37.080049 4883 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:43:37.080252 4883 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:43:37.077219 4624 master_runner.cc:186] Error getting master registration for 127.3.240.254:45507: OK, instance_id { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" instance_seqno: 1786362215679085 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:43:37.089761 4624 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 11:43:37.119092 4612 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } attempt: 5
Aug 10 11:43:37 dist-test-slave-drl0 krb5kdc[4497](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362215, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260810 11:43:37.137727 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.120318 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41578 (local address 127.3.240.252:37507)
0810 11:43:37.120523 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.120531 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.120552 (+ 21us) server_negotiation.cc:407] Connection header received
0810 11:43:37.120617 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.120622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.120698 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.120796 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.121982 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.122975 (+ 993us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.125355 (+ 2380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.126194 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.126341 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.131098 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:37.131111 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:37.131115 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:37.131146 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:37.134352 (+ 3206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.135322 (+ 970us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.135327 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.135331 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.135839 (+ 508us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.136297 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.136302 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.136305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.136572 (+ 267us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:37.136736 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:37.137144 (+ 408us) server_negotiation.cc:299] Negotiation successful
0810 11:43:37.137365 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260810 11:43:37.137753 4615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.119884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:37507 (local address 127.0.0.1:41578)
0810 11:43:37.120090 (+ 206us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:37.120113 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:37.120248 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:37.121054 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:37.121059 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:37.121612 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:37.121824 (+ 212us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:37.121836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.123106 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.123113 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:37.125166 (+ 2053us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:37.125190 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.126362 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.126369 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:37.126605 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.127396 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:37.127425 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:37.130869 (+ 3444us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:37.134602 (+ 3733us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:37.134609 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:37.134614 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:37.135189 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:37.135984 (+ 795us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:37.135991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:37.135996 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:37.136188 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:37.136710 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:37.136715 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:37.136875 (+ 160us) client_negotiation.cc:769] Sending connection context
0810 11:43:37.137028 (+ 153us) client_negotiation.cc:240] Negotiation successful
0810 11:43:37.137311 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20260810 11:43:37.145856 4612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf: Bootstrap starting.
W20260810 11:43:37.147142 4755 tablet.cc:2369] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:43:37.151363 4612 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:37.152702 4612 log.cc:824] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:37.156383 4612 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf: No bootstrap required, opened a new log
I20260810 11:43:37.165213 4612 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.165606 4612 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e6786908867457a8ce9191ccbe08fbf, State: Initialized, Role: FOLLOWER
I20260810 11:43:37.166256 4612 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.167704 4890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.168401 4890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.169615 4612 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:37.169690 4576 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 11:43:37.169914 4576 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:37.174484 4576 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.178112 4679 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6786908867457a8ce9191ccbe08fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "8766dd91e26f49558745c4772318b2e9"
I20260810 11:43:37.178552 4781 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9e6786908867457a8ce9191ccbe08fbf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "19727e15def443b69d7c787b86ff9ed1"
I20260810 11:43:37.178807 4679 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:37.179198 4781 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:37.183120 4576 leader_election.cc:288] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [CANDIDATE]: Term 1 election: Requested vote from peers 8766dd91e26f49558745c4772318b2e9 (127.3.240.253:45917), 19727e15def443b69d7c787b86ff9ed1 (127.3.240.252:37507)
I20260810 11:43:37.183521 4679 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6786908867457a8ce9191ccbe08fbf in term 1.
I20260810 11:43:37.184496 4781 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9e6786908867457a8ce9191ccbe08fbf in term 1.
I20260810 11:43:37.184537 4541 leader_election.cc:302] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8766dd91e26f49558745c4772318b2e9, 9e6786908867457a8ce9191ccbe08fbf; no voters:
I20260810 11:43:37.187129 4890 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:37.198328 4890 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [term 1 LEADER]: Becoming Leader. State: Replica: 9e6786908867457a8ce9191ccbe08fbf, State: Running, Role: LEADER
I20260810 11:43:37.199120 4890 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } }
I20260810 11:43:37.207767 4898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6786908867457a8ce9191ccbe08fbf. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.208479 4898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:37.215191 4905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
W20260810 11:43:37.268589 4652 tablet.cc:2369] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:43:37.271869 4905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:37.315038 4923 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 11:43:37.334134 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.252437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53734 (local address 127.3.240.253:45917)
0810 11:43:37.252748 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.252756 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.272946 (+ 20190us) server_negotiation.cc:407] Connection header received
0810 11:43:37.279391 (+ 6445us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.279403 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.279528 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.279623 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.294079 (+ 14456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.295513 (+ 1434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.319074 (+ 23561us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.319779 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.319915 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.324757 (+ 4842us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:37.324778 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:37.324782 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:37.324843 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:37.329873 (+ 5030us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.331100 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.331110 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.331115 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.331282 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.331911 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.331917 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.331923 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.332399 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:37.332676 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:37.333417 (+ 741us) server_negotiation.cc:299] Negotiation successful
0810 11:43:37.333685 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260810 11:43:37.340929 4679 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [term 1 FOLLOWER]: Refusing update from remote peer 9e6786908867457a8ce9191ccbe08fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:43:37.341439 4781 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [term 1 FOLLOWER]: Refusing update from remote peer 9e6786908867457a8ce9191ccbe08fbf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:43:37.343106 4890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:43:37.343840 4890 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [LEADER]: Connected to new peer: Peer: permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:43:37.371750 4840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.244339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39416 (local address 127.3.240.254:45507)
0810 11:43:37.244576 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.244587 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.251650 (+ 7063us) server_negotiation.cc:407] Connection header received
0810 11:43:37.253581 (+ 1931us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.253593 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.253717 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.253969 (+ 252us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.257117 (+ 3148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.258715 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.291090 (+ 32375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.292082 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.292226 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.331016 (+ 38790us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:37.331024 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:37.331029 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:37.331081 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:37.334730 (+ 3649us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.361770 (+ 27040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.361777 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.361782 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.362006 (+ 224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.368769 (+ 6763us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.368777 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.368780 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.369393 (+ 613us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:37.369806 (+ 413us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:37.370823 (+ 1017us) server_negotiation.cc:299] Negotiation successful
0810 11:43:37.371263 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 11:43:37.371910 4836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:37.234645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41584 (local address 127.3.240.252:37507)
0810 11:43:37.234899 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.234909 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.250951 (+ 16042us) server_negotiation.cc:407] Connection header received
0810 11:43:37.262719 (+ 11768us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.262732 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.262891 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.263126 (+ 235us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.282683 (+ 19557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.284761 (+ 2078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.298163 (+ 13402us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:37.299204 (+ 1041us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:37.299354 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:37.321055 (+ 21701us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:37.321065 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:37.321076 (+ 11us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:37.321134 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:37.330039 (+ 8905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.363136 (+ 33097us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.363143 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.363147 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.363330 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:37.365688 (+ 2358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:37.365695 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:37.365699 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:37.366143 (+ 444us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:37.366580 (+ 437us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:37.371009 (+ 4429us) server_negotiation.cc:299] Negotiation successful
0810 11:43:37.371333 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260810 11:43:37.384507 4843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6786908867457a8ce9191ccbe08fbf. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.385155 4843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.394776 4921 mvcc.cc:205] Tried to move back new op lower bound from 7316939642215235584 to 7316939641655996416. Current Snapshot: MvccSnapshot[applied={T|T < 7316939642215235584}]
I20260810 11:43:37.403450 4890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.403983 4890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:37.406095 4905 catalog_manager.cc:1347] Generated new cluster ID: ce8e794c1f3a499d9104d56ced659214
I20260810 11:43:37.406210 4905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:37.412034 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e6786908867457a8ce9191ccbe08fbf. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.412382 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.421110 4843 sys_catalog.cc:455] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.421478 4843 sys_catalog.cc:458] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.436458 4898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.436836 4898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:37.440428 4867 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9e6786908867457a8ce9191ccbe08fbf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9e6786908867457a8ce9191ccbe08fbf" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 45507 } } peers { permanent_uuid: "8766dd91e26f49558745c4772318b2e9" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 45917 } } peers { permanent_uuid: "19727e15def443b69d7c787b86ff9ed1" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 37507 } } }
I20260810 11:43:37.440709 4867 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:37.459390 4905 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:37.461501 4905 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:37.519490 4905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9e6786908867457a8ce9191ccbe08fbf: Generated new TSK 0
I20260810 11:43:37.521144 4905 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 11:43:37.635712 4035 mini_kdc.cc:348] Time spent kdestroy: real 0.101s user 0.000s sys 0.034s
W20260810 11:43:37.682114 4836 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.672280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41598 (local address 127.3.240.252:37507)
0810 11:43:37.672478 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.672484 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.680453 (+ 7969us) server_negotiation.cc:407] Connection header received
0810 11:43:37.680682 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.680687 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.680767 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.680834 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.681854 (+ 1020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41598: BlockingRecv error: recv got EOF from 127.0.0.1:41598 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260810 11:43:37.683219 4840 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.668830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39426 (local address 127.3.240.254:45507)
0810 11:43:37.669365 (+ 535us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.669388 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.679834 (+ 10446us) server_negotiation.cc:407] Connection header received
0810 11:43:37.679973 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.679980 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.680079 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.680459 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.682972 (+ 2513us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39426: BlockingRecv error: recv got EOF from 127.0.0.1:39426 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260810 11:43:37.684998 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.681737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53742 (local address 127.3.240.253:45917)
0810 11:43:37.681976 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.681982 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.683176 (+ 1194us) server_negotiation.cc:407] Connection header received
0810 11:43:37.683407 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.683413 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.683515 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.683600 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.684749 (+ 1149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53742: BlockingRecv error: recv got EOF from 127.0.0.1:53742 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20260810 11:43:37.726562 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.724304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53748 (local address 127.3.240.253:45917)
0810 11:43:37.724552 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.724559 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.724864 (+ 305us) server_negotiation.cc:407] Connection header received
0810 11:43:37.725148 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.725152 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.725247 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.725365 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.726273 (+ 908us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53748: BlockingRecv error: recv got EOF from 127.0.0.1:53748 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20260810 11:43:37.734087 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.732246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.3.240.253:45917)
0810 11:43:37.732464 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.732471 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.732670 (+ 199us) server_negotiation.cc:407] Connection header received
0810 11:43:37.732892 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.732900 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.732986 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.733106 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.733861 (+ 755us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53756: BlockingRecv error: recv got EOF from 127.0.0.1:53756 (error 108)
Metrics: {"server-negotiator.queue_time_us":84}
W20260810 11:43:37.741883 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.739932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.3.240.253:45917)
0810 11:43:37.740265 (+ 333us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.740272 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.740486 (+ 214us) server_negotiation.cc:407] Connection header received
0810 11:43:37.740700 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.740705 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.740785 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.740892 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.741630 (+ 738us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53772: BlockingRecv error: recv got EOF from 127.0.0.1:53772 (error 108)
Metrics: {"server-negotiator.queue_time_us":184}
W20260810 11:43:37.751693 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.749105 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.3.240.253:45917)
0810 11:43:37.749372 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.749377 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.749627 (+ 250us) server_negotiation.cc:407] Connection header received
0810 11:43:37.749834 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.749839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.749919 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.750128 (+ 209us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.751275 (+ 1147us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53784: BlockingRecv error: recv got EOF from 127.0.0.1:53784 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
W20260810 11:43:37.765651 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.763323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53798 (local address 127.3.240.253:45917)
0810 11:43:37.763505 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.763509 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.764198 (+ 689us) server_negotiation.cc:407] Connection header received
0810 11:43:37.764421 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.764433 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.764505 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.764616 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.765382 (+ 766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53798: BlockingRecv error: recv got EOF from 127.0.0.1:53798 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 11:43:37.789337 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.786687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53814 (local address 127.3.240.253:45917)
0810 11:43:37.786908 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.786915 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.787279 (+ 364us) server_negotiation.cc:407] Connection header received
0810 11:43:37.787459 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.787465 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.787565 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.787913 (+ 348us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.789013 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53814: BlockingRecv error: recv got EOF from 127.0.0.1:53814 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260810 11:43:37.827239 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.825030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.3.240.253:45917)
0810 11:43:37.825256 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.825262 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.825561 (+ 299us) server_negotiation.cc:407] Connection header received
0810 11:43:37.825765 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.825771 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.825900 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.826045 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.826975 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53828: BlockingRecv error: recv got EOF from 127.0.0.1:53828 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
W20260810 11:43:37.897622 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:37.895050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.3.240.253:45917)
0810 11:43:37.895265 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:37.895273 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:37.895611 (+ 338us) server_negotiation.cc:407] Connection header received
0810 11:43:37.895951 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:37.895963 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:37.896140 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:37.896359 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:37.897297 (+ 938us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53838: BlockingRecv error: recv got EOF from 127.0.0.1:53838 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260810 11:43:38.032109 4841 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:38.029786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53852 (local address 127.3.240.253:45917)
0810 11:43:38.030045 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.030052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.030311 (+ 259us) server_negotiation.cc:407] Connection header received
0810 11:43:38.030398 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.030402 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.030481 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.030578 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:38.031735 (+ 1157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53852: BlockingRecv error: recv got EOF from 127.0.0.1:53852 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
I20260810 11:43:38.033751 4863 catalog_manager.cc:1259] Loaded cluster ID: ce8e794c1f3a499d9104d56ced659214
I20260810 11:43:38.033878 4863 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: loading cluster ID for follower catalog manager: success
I20260810 11:43:38.036597 4863 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: acquiring CA information for follower catalog manager: success
I20260810 11:43:38.038417 4863 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 19727e15def443b69d7c787b86ff9ed1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:43:38.082438 4883 catalog_manager.cc:1259] Loaded cluster ID: ce8e794c1f3a499d9104d56ced659214
I20260810 11:43:38.082553 4883 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: loading cluster ID for follower catalog manager: success
I20260810 11:43:38.084966 4883 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: acquiring CA information for follower catalog manager: success
I20260810 11:43:38.086666 4883 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 8766dd91e26f49558745c4772318b2e9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:43:38.299052 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:38.292343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53864 (local address 127.3.240.253:45917)
0810 11:43:38.292702 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.292710 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.293000 (+ 290us) server_negotiation.cc:407] Connection header received
0810 11:43:38.293396 (+ 396us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.293400 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.293573 (+ 173us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.293713 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:38.294183 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.295002 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.296378 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.296836 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.296974 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:38.297569 (+ 595us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:38.298358 (+ 789us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:38.298540 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:38.298623 (+ 83us) server_negotiation.cc:299] Negotiation successful
0810 11:43:38.298708 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260810 11:43:38.312885 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:38.306035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41602 (local address 127.3.240.252:37507)
0810 11:43:38.306701 (+ 666us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.306710 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.307086 (+ 376us) server_negotiation.cc:407] Connection header received
0810 11:43:38.307374 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.307378 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.307531 (+ 153us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.307720 (+ 189us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:38.308432 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.309195 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.310638 (+ 1443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.311116 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.311254 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:38.311763 (+ 509us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:38.312270 (+ 507us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:38.312453 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:38.312508 (+ 55us) server_negotiation.cc:299] Negotiation successful
0810 11:43:38.312592 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":363,"threads_started":1}
I20260810 11:43:38.328747 5037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:38.320422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39428 (local address 127.3.240.254:45507)
0810 11:43:38.321756 (+ 1334us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.321761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.321778 (+ 17us) server_negotiation.cc:407] Connection header received
0810 11:43:38.321901 (+ 123us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.321906 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.322122 (+ 216us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.323001 (+ 879us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:38.323269 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.324530 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.326311 (+ 1781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.326757 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.326899 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:38.327510 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:38.327947 (+ 437us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:38.328055 (+ 108us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:38.328202 (+ 147us) server_negotiation.cc:299] Negotiation successful
0810 11:43:38.328465 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1202,"thread_start_us":899,"threads_started":1}
I20260810 11:43:38.328747 5025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:38.320348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41618 (local address 127.3.240.252:37507)
0810 11:43:38.320554 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.320560 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.321147 (+ 587us) server_negotiation.cc:407] Connection header received
0810 11:43:38.321414 (+ 267us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.321420 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.321502 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.321569 (+ 67us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:38.322383 (+ 814us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.323755 (+ 1372us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.326205 (+ 2450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.326885 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.327063 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:38.327617 (+ 554us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:38.327750 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:38.327849 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:38.328357 (+ 508us) server_negotiation.cc:299] Negotiation successful
0810 11:43:38.328439 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 11:43:38.329211 4841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:38.321831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53866 (local address 127.3.240.253:45917)
0810 11:43:38.322117 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:38.322127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:38.322971 (+ 844us) server_negotiation.cc:407] Connection header received
0810 11:43:38.323232 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:38.323236 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:38.323352 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:38.323469 (+ 117us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:38.323965 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.324918 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.327135 (+ 2217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:38.327755 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:38.327888 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:38.328460 (+ 572us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:38.328594 (+ 134us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:38.328643 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:38.328839 (+ 196us) server_negotiation.cc:299] Negotiation successful
0810 11:43:38.328921 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260810 11:43:38.335244 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4517
I20260810 11:43:38.336356 4605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:38.457078 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4517
I20260810 11:43:38.491897 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4624
I20260810 11:43:38.493249 4718 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:38.610836 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4624
I20260810 11:43:38.644738 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 4731
I20260810 11:43:38.646577 4816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:38.788599 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 4731
2026-08-10T11:43:38Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5348 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 10 11:43:38 dist-test-slave-drl0 krb5kdc[5044](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 10 11:43:38 dist-test-slave-drl0 krb5kdc[5044](info): set up 2 sockets
Aug 10 11:43:38 dist-test-slave-drl0 krb5kdc[5044](info): commencing operation
krb5kdc: starting...
W20260810 11:43:40.203666 4035 mini_kdc.cc:121] Time spent starting KDC: real 1.364s user 0.007s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/test-user.keytab.
Aug 10 11:43:40 dist-test-slave-drl0 krb5kdc[5044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-10T11:43:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-10T11:43:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.3.240.254@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.254@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
Entry for principal kudu/127.3.240.254 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab.
I20260810 11:43:40.603636 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39571
--webserver_interface=127.3.240.254
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260810 11:43:40.989512 5065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:40.989914 5065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:40.990051 5065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:40.990200 5065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:40.997920 5065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:40.998054 5065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:40.998155 5065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:40.998253 5065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:40.998458 5065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:41.008595 5065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.254.keytab
--principal=kudu/127.3.240.254
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.254:39571
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.3.240.254
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:41.009990 5065 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:41.011375 5065 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:41.024423 5071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:41.024387 5075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:41.024669 5070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:41.025552 5065 server_base.cc:1034] running on GCE node
I20260810 11:43:41.026510 5065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:41.028002 5065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:41.029278 5065 hybrid_clock.cc:648] HybridClock initialized: now 1786362221029183 us; error 33 us; skew 500 ppm
I20260810 11:43:41.029644 5065 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:41.033056 5065 init.cc:377] Logged in from keytab as kudu/127.3.240.254@KRBTEST.COM (short username kudu)
I20260810 11:43:41.034668 5065 webserver.cc:460] Webserver started at http://127.3.240.254:34105/ using document root <none> and password file <none>
I20260810 11:43:41.035475 5065 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:41.035576 5065 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:41.035957 5065 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:41.039122 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data/instance:
uuid: "5f86c4679e0b42a0a3caad47d77a14a6"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "1e4dc278a74a877a1ef3c16b68723e01"
server_key_iv: "8a43138447a5c096460f20e1968a1ff0"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.039978 5065 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance:
uuid: "5f86c4679e0b42a0a3caad47d77a14a6"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "1e4dc278a74a877a1ef3c16b68723e01"
server_key_iv: "8a43138447a5c096460f20e1968a1ff0"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.045261 5065 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260810 11:43:41.049260 5088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:41.050710 5065 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260810 11:43:41.051321 5065 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal
uuid: "5f86c4679e0b42a0a3caad47d77a14a6"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "1e4dc278a74a877a1ef3c16b68723e01"
server_key_iv: "8a43138447a5c096460f20e1968a1ff0"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.051694 5065 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:41.083971 5065 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:41.089159 5065 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:41.089483 5065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:41.112864 5065 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.254:39571
I20260810 11:43:41.113137 5158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.254:39571 every 8 connection(s)
I20260810 11:43:41.114387 5065 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/data/info.pb
I20260810 11:43:41.117331 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5065
I20260810 11:43:41.117581 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-0/wal/instance
I20260810 11:43:41.118212 4035 external_mini_cluster.cc:1428] Setting key 3467e8528d60ad5034d9eb414258142b
I20260810 11:43:41.118475 5159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:41.125236 5159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:41.152359 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.129345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39571 (local address 127.0.0.1:49650)
0810 11:43:41.130452 (+ 1107us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.130508 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:41.133032 (+ 2524us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:41.133935 (+ 903us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:41.133941 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:41.134659 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:41.135085 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.135115 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.137006 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.137010 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.138082 (+ 1072us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.138095 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.139660 (+ 1565us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.139666 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.139817 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.140523 (+ 706us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:41.140547 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:41.143014 (+ 2467us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:41.147965 (+ 4951us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.147975 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.147985 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.148390 (+ 405us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.148948 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.148952 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.148955 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.149114 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.150237 (+ 1123us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:41.150243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:41.150445 (+ 202us) client_negotiation.cc:769] Sending connection context
0810 11:43:41.150681 (+ 236us) client_negotiation.cc:240] Negotiation successful
0810 11:43:41.151171 (+ 490us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":421,"threads_started":1}
I20260810 11:43:41.152418 5162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.130773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49650 (local address 127.3.240.254:39571)
0810 11:43:41.131762 (+ 989us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:41.131771 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:41.131843 (+ 72us) server_negotiation.cc:407] Connection header received
0810 11:43:41.133435 (+ 1592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:41.133440 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:41.133688 (+ 248us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:41.133908 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:41.135369 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.136798 (+ 1429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.138240 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.139333 (+ 1093us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.139532 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.143272 (+ 3740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:41.143285 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:41.143304 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:41.143347 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:41.147793 (+ 4446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.148557 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.148568 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.148574 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.148783 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.149291 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.149299 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.149305 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.150068 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:41.150284 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:41.150909 (+ 625us) server_negotiation.cc:299] Negotiation successful
0810 11:43:41.151387 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":91,"server-negotiator.queue_time_us":601,"thread_start_us":401,"threads_started":1}
I20260810 11:43:41.155818 5159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:41.157030 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.156620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39760)
0810 11:43:41.156794 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.156922 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 11:43:41.157377 5092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.3.240.253@KRBTEST.COM; defaulting to no policy
I20260810 11:43:41.159162 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.158661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39776)
0810 11:43:41.158900 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.159026 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260810 11:43:41.159584 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:34933: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.253@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
Entry for principal kudu/127.3.240.253 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab.
I20260810 11:43:41.203161 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34933
--webserver_interface=127.3.240.253
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:43:41.209089 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } attempt: 1
I20260810 11:43:41.210642 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.209894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39782)
0810 11:43:41.210263 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.210438 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260810 11:43:41.212011 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.211595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39798)
0810 11:43:41.211760 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.211846 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260810 11:43:41.212232 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:34933: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
I20260810 11:43:41.280634 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } attempt: 2
I20260810 11:43:41.282298 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.281668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39812)
0810 11:43:41.282001 (+ 333us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.282118 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260810 11:43:41.283505 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.283113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39814)
0810 11:43:41.283270 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.283366 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260810 11:43:41.283735 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:34933: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
I20260810 11:43:41.375082 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } attempt: 3
I20260810 11:43:41.376361 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.375874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39818)
0810 11:43:41.376101 (+ 227us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.376204 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20260810 11:43:41.377374 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.377018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39826)
0810 11:43:41.377174 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.377239 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260810 11:43:41.377799 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:34933: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
I20260810 11:43:41.521167 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } attempt: 4
I20260810 11:43:41.522758 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.521953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39834)
0810 11:43:41.522519 (+ 566us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.522619 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":226}
I20260810 11:43:41.524029 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.523532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39838)
0810 11:43:41.523721 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.523807 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260810 11:43:41.524315 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.253:34933: Network error: Client connection negotiation failed: client connection to 127.3.240.253:34933: connect: Connection refused (error 111)
W20260810 11:43:41.584510 5168 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:41.584888 5168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:41.585040 5168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:41.585137 5168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:41.592415 5168 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:41.592506 5168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:41.592564 5168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:41.592646 5168 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:41.592737 5168 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:41.605316 5168 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.253.keytab
--principal=kudu/127.3.240.253
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.253:34933
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.3.240.253
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:41.606724 5168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:41.608108 5168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:41.622968 5177 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:41.623087 5180 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:41.623037 5178 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:41.624392 5168 server_base.cc:1034] running on GCE node
I20260810 11:43:41.625231 5168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:41.627584 5168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:41.628898 5168 hybrid_clock.cc:648] HybridClock initialized: now 1786362221628820 us; error 74 us; skew 500 ppm
I20260810 11:43:41.629379 5168 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:41.636453 5168 init.cc:377] Logged in from keytab as kudu/127.3.240.253@KRBTEST.COM (short username kudu)
I20260810 11:43:41.638312 5168 webserver.cc:460] Webserver started at http://127.3.240.253:33109/ using document root <none> and password file <none>
I20260810 11:43:41.639194 5168 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:41.639492 5168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:41.640142 5168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:41.644235 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data/instance:
uuid: "efc440c0e71c4e05ab4d7b90689c7991"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "f3c199a1aa5283a87d57f1875e1c1577"
server_key_iv: "8a5eef60ee949afc65afd6240e57c68d"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.645190 5168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal/instance:
uuid: "efc440c0e71c4e05ab4d7b90689c7991"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "f3c199a1aa5283a87d57f1875e1c1577"
server_key_iv: "8a5eef60ee949afc65afd6240e57c68d"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.651086 5168 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260810 11:43:41.654786 5192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:41.656996 5168 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260810 11:43:41.657202 5168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal
uuid: "efc440c0e71c4e05ab4d7b90689c7991"
format_stamp: "Formatted at 2026-08-10 11:43:41 on dist-test-slave-drl0"
server_key: "f3c199a1aa5283a87d57f1875e1c1577"
server_key_iv: "8a5eef60ee949afc65afd6240e57c68d"
server_key_version: "encryptionkey@0"
I20260810 11:43:41.657446 5168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:41.680552 5168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:41.685163 5168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:41.685397 5168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:41.706575 5168 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.253:34933
I20260810 11:43:41.706655 5266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.253:34933 every 8 connection(s)
I20260810 11:43:41.708034 5168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/data/info.pb
I20260810 11:43:41.712971 5267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:41.713621 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5168
I20260810 11:43:41.713878 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-1/wal/instance
I20260810 11:43:41.714572 4035 external_mini_cluster.cc:1428] Setting key d9ebb38b8078a982577ddbad74363f5d
I20260810 11:43:41.719964 5267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:41.745354 5269 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.724085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49662 (local address 127.3.240.254:39571)
0810 11:43:41.724590 (+ 505us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:41.724597 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:41.726553 (+ 1956us) server_negotiation.cc:407] Connection header received
0810 11:43:41.729698 (+ 3145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:41.729705 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:41.729840 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:41.729993 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:41.732048 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.732769 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.734291 (+ 1522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.734712 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.734856 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.738652 (+ 3796us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:41.738658 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:41.738661 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:41.738706 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:41.742017 (+ 3311us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.742839 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.742851 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.742854 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.743047 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.743503 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.743508 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.743511 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.743799 (+ 288us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:41.743998 (+ 199us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:41.744717 (+ 719us) server_negotiation.cc:299] Negotiation successful
0810 11:43:41.744949 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":764,"threads_started":1}
I20260810 11:43:41.746520 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.724650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39571 (local address 127.0.0.1:49662)
0810 11:43:41.726370 (+ 1720us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.726426 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:41.729244 (+ 2818us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:41.730446 (+ 1202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:41.730465 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:41.731196 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:41.731782 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.731809 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.732951 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.732960 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.734131 (+ 1171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.734150 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.734984 (+ 834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.734993 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.735214 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.735915 (+ 701us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:41.735938 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:41.738401 (+ 2463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:41.742187 (+ 3786us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.742198 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.742217 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.742696 (+ 479us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.743167 (+ 471us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.743172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.743176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.743368 (+ 192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.743957 (+ 589us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:41.743965 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:41.744336 (+ 371us) client_negotiation.cc:769] Sending connection context
0810 11:43:41.744621 (+ 285us) client_negotiation.cc:240] Negotiation successful
0810 11:43:41.745187 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1410,"thread_start_us":1180,"threads_started":1}
I20260810 11:43:41.748430 5267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.3.240.252@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.3.240.252@KRBTEST.COM" created.
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:41.764925 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.749268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39854)
0810 11:43:41.749463 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.749493 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:41.749581 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:41.750746 (+ 1165us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:41.750751 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:41.751237 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:41.751375 (+ 138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.751383 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.752784 (+ 1401us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.752791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.753695 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.753707 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.754644 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.754650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.754847 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.755498 (+ 651us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:41.755515 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:41.757976 (+ 2461us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:41.762381 (+ 4405us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.762388 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.762392 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.762760 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.763192 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.763199 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.763202 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.763353 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.763939 (+ 586us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:41.763946 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:41.764084 (+ 138us) client_negotiation.cc:769] Sending connection context
0810 11:43:41.764236 (+ 152us) client_negotiation.cc:240] Negotiation successful
0810 11:43:41.764503 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260810 11:43:41.764973 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.749599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39854 (local address 127.3.240.253:34933)
0810 11:43:41.750331 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:41.750342 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:41.750368 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:43:41.750412 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:41.750418 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:41.750592 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:41.750755 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:41.751563 (+ 808us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.752646 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.753841 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.754490 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.754623 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.758298 (+ 3675us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:41.758311 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:41.758326 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:41.758355 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:41.762239 (+ 3884us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.762903 (+ 664us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.762911 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.762915 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.763085 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.763458 (+ 373us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.763465 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.763469 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.763813 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:41.764039 (+ 226us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:41.764395 (+ 356us) server_negotiation.cc:299] Negotiation successful
0810 11:43:41.764605 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":572,"thread_start_us":435,"threads_started":1}
I20260810 11:43:41.768287 5267 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:41.769346 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.768896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33174)
0810 11:43:41.769074 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.769193 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 11:43:41.769577 5195 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:41.771198 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.770818 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33182)
0810 11:43:41.771025 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.771099 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260810 11:43:41.771413 5267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
Entry for principal kudu/127.3.240.252 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab.
I20260810 11:43:41.803694 4035 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskagqWOB/build/asan/bin/kudu
/tmp/dist-test-taskagqWOB/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39759
--webserver_interface=127.3.240.252
--webserver_port=0
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.3.240.212:42595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260810 11:43:41.820793 5267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 1
I20260810 11:43:41.822547 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.821538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33188)
0810 11:43:41.821998 (+ 460us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.822149 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":198}
I20260810 11:43:41.823606 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } attempt: 5
I20260810 11:43:41.823899 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.823372 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33190)
0810 11:43:41.823631 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.823741 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260810 11:43:41.824193 5267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Aug 10 11:43:41 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:41.839994 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.824250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39856)
0810 11:43:41.824466 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.824489 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:41.824651 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:41.825025 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:41.825033 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:41.825590 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:41.825866 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.825882 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.827048 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.827057 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.828741 (+ 1684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:41.828761 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.829629 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.829640 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:41.830097 (+ 457us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.830949 (+ 852us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:41.830981 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:41.833583 (+ 2602us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:41.837423 (+ 3840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.837430 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.837438 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.837840 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.838261 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:41.838266 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:41.838270 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:41.838414 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:41.838988 (+ 574us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:41.838994 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:41.839141 (+ 147us) client_negotiation.cc:769] Sending connection context
0810 11:43:41.839305 (+ 164us) client_negotiation.cc:240] Negotiation successful
0810 11:43:41.839632 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260810 11:43:41.839982 5272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.824478 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39856 (local address 127.3.240.253:34933)
0810 11:43:41.824645 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:41.824651 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:41.824679 (+ 28us) server_negotiation.cc:407] Connection header received
0810 11:43:41.824793 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:41.824796 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:41.824871 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:41.824955 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:41.826046 (+ 1091us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.826859 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.828992 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:41.829435 (+ 443us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:41.829546 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:41.833791 (+ 4245us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:41.833796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:41.833800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:41.833836 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:41.837235 (+ 3399us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.837991 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.837999 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.838007 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.838125 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:41.838531 (+ 406us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:41.838537 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:41.838540 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:41.838833 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:41.839020 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:41.839406 (+ 386us) server_negotiation.cc:299] Negotiation successful
0810 11:43:41.839632 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20260810 11:43:41.841195 5159 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:41.842142 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.841772 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33194)
0810 11:43:41.841936 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.842016 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260810 11:43:41.843472 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.843139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33206)
0810 11:43:41.843296 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.843379 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260810 11:43:41.843673 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:41.892472 5267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 2
I20260810 11:43:41.893790 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.893232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33218)
0810 11:43:41.893478 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.893593 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94}
I20260810 11:43:41.894906 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 1
I20260810 11:43:41.895192 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.894762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33234)
0810 11:43:41.894969 (+ 207us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.895060 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
W20260810 11:43:41.895399 5267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:41.896006 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.895520 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33246)
0810 11:43:41.895717 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.895816 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20260810 11:43:41.897104 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.896717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33252)
0810 11:43:41.896879 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.896967 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260810 11:43:41.897524 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:41.965951 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 2
I20260810 11:43:41.967293 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.966647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33264)
0810 11:43:41.966910 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.967060 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260810 11:43:41.968474 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.968066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33274)
0810 11:43:41.968226 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.968330 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
W20260810 11:43:41.968736 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:41.986675 5267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 3
I20260810 11:43:41.987949 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.987454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33278)
0810 11:43:41.987683 (+ 229us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.987784 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20260810 11:43:41.988997 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:41.988596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33286)
0810 11:43:41.988830 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:41.988901 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
W20260810 11:43:41.989192 5267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:42.075012 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 3
I20260810 11:43:42.076785 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.075959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33296)
0810 11:43:42.076328 (+ 369us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.076493 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":134}
I20260810 11:43:42.078327 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.077698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33302)
0810 11:43:42.077977 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.078106 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
W20260810 11:43:42.078593 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:42.132474 5267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 4
I20260810 11:43:42.134126 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.133424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33316)
0810 11:43:42.133822 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.133952 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":196}
I20260810 11:43:42.135319 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.134995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33318)
0810 11:43:42.135149 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.135216 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260810 11:43:42.135586 5267 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
W20260810 11:43:42.198889 5278 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260810 11:43:42.199318 5278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260810 11:43:42.199420 5278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260810 11:43:42.199582 5278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260810 11:43:42.206995 5278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260810 11:43:42.207078 5278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260810 11:43:42.207160 5278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260810 11:43:42.207245 5278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260810 11:43:42.207352 5278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260810 11:43:42.217248 5278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.3.240.212:42595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.3.240.254:39571,127.3.240.253:34933,127.3.240.252:39759
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/krb5kdc/kudu_127.3.240.252.keytab
--principal=kudu/127.3.240.252
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.3.240.252:39759
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.3.240.252
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 10 Aug 2026 10:32:47 UTC on ccycloud.quasar-nosthy.root.comops.site
build id 4577509
ASAN enabled
I20260810 11:43:42.218482 5278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260810 11:43:42.219854 5278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260810 11:43:42.231680 5289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:42.231727 5290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260810 11:43:42.231890 5298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260810 11:43:42.232811 5278 server_base.cc:1034] running on GCE node
I20260810 11:43:42.233549 5278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260810 11:43:42.234838 5278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260810 11:43:42.236123 5278 hybrid_clock.cc:648] HybridClock initialized: now 1786362222235991 us; error 60 us; skew 500 ppm
I20260810 11:43:42.236521 5278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362222, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260810 11:43:42.240010 5278 init.cc:377] Logged in from keytab as kudu/127.3.240.252@KRBTEST.COM (short username kudu)
I20260810 11:43:42.241526 5278 webserver.cc:460] Webserver started at http://127.3.240.252:39723/ using document root <none> and password file <none>
I20260810 11:43:42.242213 5278 fs_manager.cc:359] Metadata directory not provided
I20260810 11:43:42.242400 5278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260810 11:43:42.242725 5278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260810 11:43:42.247215 5278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data/instance:
uuid: "c5d2b79be6a14f6eaad8cee88426a300"
format_stamp: "Formatted at 2026-08-10 11:43:42 on dist-test-slave-drl0"
server_key: "78f5de32ad7954e9e1bc850d80d6fa18"
server_key_iv: "0c15090ee6687280088520b54c235a03"
server_key_version: "encryptionkey@0"
I20260810 11:43:42.248281 5278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal/instance:
uuid: "c5d2b79be6a14f6eaad8cee88426a300"
format_stamp: "Formatted at 2026-08-10 11:43:42 on dist-test-slave-drl0"
server_key: "78f5de32ad7954e9e1bc850d80d6fa18"
server_key_iv: "0c15090ee6687280088520b54c235a03"
server_key_version: "encryptionkey@0"
I20260810 11:43:42.255403 5278 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260810 11:43:42.255811 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 4
I20260810 11:43:42.257007 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.256570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33334)
0810 11:43:42.256768 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.256857 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
I20260810 11:43:42.258304 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.257698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33342)
0810 11:43:42.257994 (+ 296us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.258126 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":136}
W20260810 11:43:42.258646 5159 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.3.240.252:39759: Network error: Client connection negotiation failed: client connection to 127.3.240.252:39759: connect: Connection refused (error 111)
I20260810 11:43:42.260646 5306 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260810 11:43:42.262709 5278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260810 11:43:42.262903 5278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data,/tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal
uuid: "c5d2b79be6a14f6eaad8cee88426a300"
format_stamp: "Formatted at 2026-08-10 11:43:42 on dist-test-slave-drl0"
server_key: "78f5de32ad7954e9e1bc850d80d6fa18"
server_key_iv: "0c15090ee6687280088520b54c235a03"
server_key_version: "encryptionkey@0"
I20260810 11:43:42.263279 5278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260810 11:43:42.280258 5278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260810 11:43:42.285005 5278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260810 11:43:42.285247 5278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260810 11:43:42.307559 5278 rpc_server.cc:307] RPC server started. Bound to: 127.3.240.252:39759
I20260810 11:43:42.307602 5381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.3.240.252:39759 every 8 connection(s)
I20260810 11:43:42.309407 5278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/data/info.pb
I20260810 11:43:42.313651 5382 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260810 11:43:42.317318 4035 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskagqWOB/build/asan/bin/kudu as pid 5278
I20260810 11:43:42.317540 4035 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskagqWOB/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786361809851056-4035-0/minicluster-data/master-2/wal/instance
I20260810 11:43:42.318100 4035 external_mini_cluster.cc:1428] Setting key 52dff41887537ec3cb96af27aafcd032
I20260810 11:43:42.321928 5382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362222, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:42.344066 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.325650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49678 (local address 127.3.240.254:39571)
0810 11:43:42.326204 (+ 554us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.326210 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.326647 (+ 437us) server_negotiation.cc:407] Connection header received
0810 11:43:42.326866 (+ 219us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.326915 (+ 49us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.327029 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.327149 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.328610 (+ 1461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.329713 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.332400 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.332929 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.333040 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.336669 (+ 3629us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.336676 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.336681 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.336731 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.340539 (+ 3808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.341261 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.341335 (+ 74us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.341338 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.341591 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.342232 (+ 641us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.342236 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.342239 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.342607 (+ 368us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.342784 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.343484 (+ 700us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.343726 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":418,"thread_start_us":284,"threads_started":1}
I20260810 11:43:42.345141 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.325717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.254:39571 (local address 127.0.0.1:49678)
0810 11:43:42.326513 (+ 796us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.326551 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:42.326730 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:42.327212 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:42.327218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:42.328075 (+ 857us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:42.328416 (+ 341us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.328430 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.330086 (+ 1656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.330094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.332126 (+ 2032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.332145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.333200 (+ 1055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.333211 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.333419 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.334136 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:42.334159 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:42.336469 (+ 2310us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:42.340661 (+ 4192us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.340673 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.340685 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.341104 (+ 419us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.341747 (+ 643us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.341750 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.341752 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.342079 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.342779 (+ 700us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:42.342787 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:42.343041 (+ 254us) client_negotiation.cc:769] Sending connection context
0810 11:43:42.343342 (+ 301us) client_negotiation.cc:240] Negotiation successful
0810 11:43:42.343841 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"mutex_wait_us":193,"thread_start_us":451,"threads_started":1}
I20260810 11:43:42.345161 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.320103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33350 (local address 127.3.240.252:39759)
0810 11:43:42.320979 (+ 876us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.320991 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.321034 (+ 43us) server_negotiation.cc:407] Connection header received
0810 11:43:42.322625 (+ 1591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.322647 (+ 22us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.323122 (+ 475us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.323445 (+ 323us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.325326 (+ 1881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.327388 (+ 2062us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.329050 (+ 1662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.330174 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.330314 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.336316 (+ 6002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.336327 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.336339 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.336376 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.340142 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.341222 (+ 1080us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.341231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.341235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.341415 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.341997 (+ 582us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.342002 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.342005 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.342564 (+ 559us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.342845 (+ 281us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.343265 (+ 420us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.343711 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":8,"server-negotiator.queue_time_us":660,"thread_start_us":392,"threads_started":1}
I20260810 11:43:42.347164 5382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } has no permanent_uuid. Determining permanent_uuid...
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362222, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.254@KRBTEST.COM
I20260810 11:43:42.363266 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.347876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.253:34933 (local address 127.0.0.1:39862)
0810 11:43:42.348047 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.348066 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:42.348171 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:42.349378 (+ 1207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:42.349382 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:42.349753 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:42.349905 (+ 152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.349914 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.350916 (+ 1002us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.350924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.351821 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.351833 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.352529 (+ 696us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.352535 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.352736 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.353276 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:42.353293 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:42.355968 (+ 2675us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:42.360212 (+ 4244us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.360218 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.360222 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.360741 (+ 519us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.361251 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.361258 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.361264 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.361450 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.362126 (+ 676us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:42.362132 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:42.362305 (+ 173us) client_negotiation.cc:769] Sending connection context
0810 11:43:42.362466 (+ 161us) client_negotiation.cc:240] Negotiation successful
0810 11:43:42.362848 (+ 382us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43}
I20260810 11:43:42.363312 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.348003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39862 (local address 127.3.240.253:34933)
0810 11:43:42.349093 (+ 1090us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.349097 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.349112 (+ 15us) server_negotiation.cc:407] Connection header received
0810 11:43:42.349161 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.349165 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.349226 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.349361 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.350094 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.350777 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.351975 (+ 1198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.352383 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.352520 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.356189 (+ 3669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.356194 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.356197 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.356227 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.360063 (+ 3836us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.360900 (+ 837us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.360905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.360907 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.361104 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.361545 (+ 441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.361553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.361558 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.361980 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.362165 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.362636 (+ 471us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.363021 (+ 385us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":900,"threads_started":1}
I20260810 11:43:42.364224 5382 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } has no permanent_uuid. Determining permanent_uuid...
I20260810 11:43:42.364619 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.348447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49688 (local address 127.3.240.254:39571)
0810 11:43:42.348611 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.348615 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.348687 (+ 72us) server_negotiation.cc:407] Connection header received
0810 11:43:42.348734 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.348737 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.348800 (+ 63us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.348917 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.350055 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.350741 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.352029 (+ 1288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.352460 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.352596 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.357458 (+ 4862us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.357464 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.357467 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.357497 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.361204 (+ 3707us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.362123 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.362128 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.362132 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.362294 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.362902 (+ 608us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.362910 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.362916 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.363288 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.363585 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.364087 (+ 502us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.364326 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362222, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.252@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.3.240.253@KRBTEST.COM
I20260810 11:43:42.379612 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.365057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33364 (local address 127.3.240.252:39759)
0810 11:43:42.365234 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.365240 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.365256 (+ 16us) server_negotiation.cc:407] Connection header received
0810 11:43:42.365299 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.365303 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.365367 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.365457 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.366164 (+ 707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.367160 (+ 996us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.368312 (+ 1152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.368715 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.368867 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.372872 (+ 4005us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.372877 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.372880 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.372907 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.376466 (+ 3559us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.377505 (+ 1039us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.377512 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.377517 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.377694 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.378190 (+ 496us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.378194 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.378196 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.378523 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.378705 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.379177 (+ 472us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.379369 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":34,"server-negotiator.queue_time_us":53}
I20260810 11:43:42.380020 5389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.364909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33364)
0810 11:43:42.365072 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.365093 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:42.365197 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:42.365516 (+ 319us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:42.365522 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:42.365896 (+ 374us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:42.366023 (+ 127us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.366031 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.367278 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.367286 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.368171 (+ 885us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.368193 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.368874 (+ 681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.368879 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.369132 (+ 253us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.369956 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:42.369984 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:42.372676 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:42.376764 (+ 4088us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.376771 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.376774 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.377314 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.377852 (+ 538us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.377858 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.377861 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.378040 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.378683 (+ 643us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:42.378691 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:42.378874 (+ 183us) client_negotiation.cc:769] Sending connection context
0810 11:43:42.379098 (+ 224us) client_negotiation.cc:240] Negotiation successful
0810 11:43:42.379584 (+ 486us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":177}
I20260810 11:43:42.383154 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.366332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39872 (local address 127.3.240.253:34933)
0810 11:43:42.366490 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.366495 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.366513 (+ 18us) server_negotiation.cc:407] Connection header received
0810 11:43:42.366647 (+ 134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.366653 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.366720 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.366815 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.368359 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.369187 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.370886 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.371359 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.371500 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.376833 (+ 5333us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.376842 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.376850 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.376889 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.380346 (+ 3457us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.381055 (+ 709us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.381062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.381065 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.381199 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.381608 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.381615 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.381621 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.381991 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.382286 (+ 295us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.382558 (+ 272us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.382780 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260810 11:43:42.386294 5382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: Bootstrap starting.
I20260810 11:43:42.388530 4035 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260810 11:43:42.390460 5382 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:42.391796 5382 log.cc:824] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:42.394634 5382 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: No bootstrap required, opened a new log
I20260810 11:43:42.391721 5128 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "5f86c4679e0b42a0a3caad47d77a14a6"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49688
I20260810 11:43:42.401974 5382 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.402285 5382 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5d2b79be6a14f6eaad8cee88426a300, State: Initialized, Role: FOLLOWER
I20260810 11:43:42.402769 5382 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.403823 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.404342 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.405604 5382 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 11:43:42.415608 5410 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:43:42.415745 5410 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:43:42.413646 5278 master_runner.cc:186] Error getting master registration for 127.3.240.254:39571: OK, instance_id { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" instance_seqno: 1786362221107197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:43:42.426631 5278 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 11:43:42.434849 5267 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 5
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.253@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260810 11:43:42.453236 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.435832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33366 (local address 127.3.240.252:39759)
0810 11:43:42.436057 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.436064 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.436086 (+ 22us) server_negotiation.cc:407] Connection header received
0810 11:43:42.436151 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.436159 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.436242 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.436366 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.437464 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.438666 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.440443 (+ 1777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.441167 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.441299 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.445617 (+ 4318us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.445638 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.445645 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.445692 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.449597 (+ 3905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.450720 (+ 1123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.450729 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.450733 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.450878 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.451538 (+ 660us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.451545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.451551 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.452000 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.452172 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.452621 (+ 449us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.452848 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260810 11:43:42.453225 5270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.435616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33366)
0810 11:43:42.435807 (+ 191us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.435827 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:42.435987 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:42.436544 (+ 557us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:42.436551 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:42.437072 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:42.437277 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.437289 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.438823 (+ 1534us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.438828 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.440267 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.440282 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.441350 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.441358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.441624 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.442354 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:42.442378 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:42.445362 (+ 2984us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:42.449961 (+ 4599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.449968 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.449972 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.450562 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.451219 (+ 657us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.451225 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.451230 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.451407 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.452140 (+ 733us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:42.452146 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:42.452322 (+ 176us) client_negotiation.cc:769] Sending connection context
0810 11:43:42.452511 (+ 189us) client_negotiation.cc:240] Negotiation successful
0810 11:43:42.452822 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":54}
I20260810 11:43:42.460218 5267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: Bootstrap starting.
I20260810 11:43:42.464748 5267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:42.466090 5267 log.cc:824] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:42.470669 5267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: No bootstrap required, opened a new log
I20260810 11:43:42.481755 5267 raft_consensus.cc:348] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.482191 5267 raft_consensus.cc:729] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: efc440c0e71c4e05ab4d7b90689c7991, State: Initialized, Role: FOLLOWER
I20260810 11:43:42.482756 5267 consensus_queue.cc:260] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.483837 5416 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.484229 5416 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.485121 5267 sys_catalog.cc:564] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: configured and running, proceeding with master startup.
W20260810 11:43:42.501829 5429 catalog_manager.cc:1558] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260810 11:43:42.501968 5429 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260810 11:43:42.497977 5168 master_runner.cc:186] Error getting master registration for 127.3.240.254:39571: OK, instance_id { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" instance_seqno: 1786362221107197 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260810 11:43:42.510149 5168 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260810 11:43:42.535952 5159 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } attempt: 5
W20260810 11:43:42.537580 5313 tablet.cc:2369] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 10 11:43:42 dist-test-slave-drl0 krb5kdc[5044](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786362221, etypes {rep=17 tkt=17 ses=17}, kudu/127.3.240.254@KRBTEST.COM for kudu/127.3.240.252@KRBTEST.COM
I20260810 11:43:42.555812 5161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.537120 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.3.240.252:39759 (local address 127.0.0.1:33380)
0810 11:43:42.537612 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0810 11:43:42.537645 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0810 11:43:42.537833 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0810 11:43:42.538547 (+ 714us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0810 11:43:42.538553 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0810 11:43:42.539271 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0810 11:43:42.539532 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.539548 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.541492 (+ 1944us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.541501 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.543072 (+ 1571us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0810 11:43:42.543093 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.544176 (+ 1083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.544185 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0810 11:43:42.544406 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.545180 (+ 774us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0810 11:43:42.545218 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0810 11:43:42.548193 (+ 2975us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0810 11:43:42.552775 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.552781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.552786 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.553325 (+ 539us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.553787 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0810 11:43:42.553793 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0810 11:43:42.553797 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0810 11:43:42.554003 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0810 11:43:42.554689 (+ 686us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0810 11:43:42.554697 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0810 11:43:42.554887 (+ 190us) client_negotiation.cc:769] Sending connection context
0810 11:43:42.555073 (+ 186us) client_negotiation.cc:240] Negotiation successful
0810 11:43:42.555368 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":224}
I20260810 11:43:42.555939 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.537417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33380 (local address 127.3.240.252:39759)
0810 11:43:42.537769 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.537776 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.537802 (+ 26us) server_negotiation.cc:407] Connection header received
0810 11:43:42.538024 (+ 222us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.538031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.538153 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.538270 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.539775 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.541225 (+ 1450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.543263 (+ 2038us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.543988 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.544137 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.548400 (+ 4263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.548407 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.548413 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.548453 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.552617 (+ 4164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.553468 (+ 851us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.553475 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.553479 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.553657 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.554152 (+ 495us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.554158 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.554161 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.554552 (+ 391us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.554716 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.555227 (+ 511us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.555478 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180}
I20260810 11:43:42.562855 5159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: Bootstrap starting.
I20260810 11:43:42.568110 5159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: Neither blocks nor log segments found. Creating new log.
I20260810 11:43:42.569862 5159 log.cc:824] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: Log is configured to *not* fsync() on all Append() calls
I20260810 11:43:42.573803 5159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: No bootstrap required, opened a new log
I20260810 11:43:42.581780 5159 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.582191 5159 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f86c4679e0b42a0a3caad47d77a14a6, State: Initialized, Role: FOLLOWER
I20260810 11:43:42.582893 5159 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.584710 5432 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.585381 5432 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.585698 5159 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260810 11:43:42.585793 5128 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260810 11:43:42.585995 5128 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:42.589685 5128 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.592347 5233 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "efc440c0e71c4e05ab4d7b90689c7991"
I20260810 11:43:42.592768 5346 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "c5d2b79be6a14f6eaad8cee88426a300"
I20260810 11:43:42.593029 5233 raft_consensus.cc:3037] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:42.593214 5346 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [term 0 FOLLOWER]: Advancing to term 1
I20260810 11:43:42.596521 5346 raft_consensus.cc:2454] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f86c4679e0b42a0a3caad47d77a14a6 in term 1.
I20260810 11:43:42.597146 5092 leader_election.cc:302] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5f86c4679e0b42a0a3caad47d77a14a6, c5d2b79be6a14f6eaad8cee88426a300; no voters:
I20260810 11:43:42.597640 5128 leader_election.cc:288] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [CANDIDATE]: Term 1 election: Requested vote from peers efc440c0e71c4e05ab4d7b90689c7991 (127.3.240.253:34933), c5d2b79be6a14f6eaad8cee88426a300 (127.3.240.252:39759)
I20260810 11:43:42.597678 5233 raft_consensus.cc:2454] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5f86c4679e0b42a0a3caad47d77a14a6 in term 1.
W20260810 11:43:42.598471 5095 tablet.cc:2369] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:43:42.604894 5432 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260810 11:43:42.615188 5432 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [term 1 LEADER]: Becoming Leader. State: Replica: 5f86c4679e0b42a0a3caad47d77a14a6, State: Running, Role: LEADER
I20260810 11:43:42.615832 5432 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } }
I20260810 11:43:42.627079 5439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f86c4679e0b42a0a3caad47d77a14a6. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.627532 5439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:42.640494 5449 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260810 11:43:42.670524 5449 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260810 11:43:42.682539 5460 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260810 11:43:42.687064 5388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.632034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49704 (local address 127.3.240.254:39571)
0810 11:43:42.632213 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.632220 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.644438 (+ 12218us) server_negotiation.cc:407] Connection header received
0810 11:43:42.645861 (+ 1423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.645867 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.645955 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.646031 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.654060 (+ 8029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.655320 (+ 1260us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.657565 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.658255 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.658355 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.669086 (+ 10731us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.669093 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.669099 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.669144 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.672797 (+ 3653us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.677801 (+ 5004us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.677809 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.677815 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.678000 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.678748 (+ 748us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.678754 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.678757 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.679195 (+ 438us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.679458 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.685805 (+ 6347us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.686156 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
W20260810 11:43:42.688410 5200 tablet.cc:2369] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260810 11:43:42.691253 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.633142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39882 (local address 127.3.240.253:34933)
0810 11:43:42.633321 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.633327 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.647430 (+ 14103us) server_negotiation.cc:407] Connection header received
0810 11:43:42.648613 (+ 1183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.648619 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.648703 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.648774 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.656118 (+ 7344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.657304 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.662867 (+ 5563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.663571 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.663677 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.680350 (+ 16673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.680357 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.680362 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.680412 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.684659 (+ 4247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.688399 (+ 3740us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.688406 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.688409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.688598 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.689123 (+ 525us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.689129 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.689133 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.689547 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.689866 (+ 319us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.690460 (+ 594us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.690820 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260810 11:43:42.698833 5387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:42.645023 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33390 (local address 127.3.240.252:39759)
0810 11:43:42.645244 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:42.645250 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:42.648058 (+ 2808us) server_negotiation.cc:407] Connection header received
0810 11:43:42.650645 (+ 2587us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:42.650653 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:42.650742 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:42.650829 (+ 87us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:42.659196 (+ 8367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.660325 (+ 1129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.671062 (+ 10737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:42.671793 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:42.671901 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:42.679093 (+ 7192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0810 11:43:42.679102 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0810 11:43:42.679109 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0810 11:43:42.679152 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0810 11:43:42.683577 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.695142 (+ 11565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.695149 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.695152 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.695324 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0810 11:43:42.695811 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0810 11:43:42.695817 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0810 11:43:42.695821 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0810 11:43:42.696307 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0810 11:43:42.696569 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:42.698098 (+ 1529us) server_negotiation.cc:299] Negotiation successful
0810 11:43:42.698396 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260810 11:43:42.712162 5233 raft_consensus.cc:1261] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [term 1 FOLLOWER]: Refusing update from remote peer 5f86c4679e0b42a0a3caad47d77a14a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:43:42.713255 5432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:43:42.717070 5346 raft_consensus.cc:1261] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [term 1 FOLLOWER]: Refusing update from remote peer 5f86c4679e0b42a0a3caad47d77a14a6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260810 11:43:42.718297 5432 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260810 11:43:42.759356 5416 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f86c4679e0b42a0a3caad47d77a14a6. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.759689 5416 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.807317 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f86c4679e0b42a0a3caad47d77a14a6. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.807718 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.808869 5439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.809161 5439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:42.810810 5439 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.811051 5439 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6 [sys.catalog]: This master's current role is: LEADER
I20260810 11:43:42.811517 5465 mvcc.cc:205] Tried to move back new op lower bound from 7316939664214376448 to 7316939663852781568. Current Snapshot: MvccSnapshot[applied={T|T < 7316939664214376448}]
I20260810 11:43:42.816730 5449 catalog_manager.cc:1347] Generated new cluster ID: 5fc4ed2fd3ca4427a9b5f4d4bfaec402
I20260810 11:43:42.816843 5449 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260810 11:43:42.831378 5416 sys_catalog.cc:455] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.831658 5416 sys_catalog.cc:458] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.832613 5393 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5f86c4679e0b42a0a3caad47d77a14a6" member_type: VOTER last_known_addr { host: "127.3.240.254" port: 39571 } } peers { permanent_uuid: "efc440c0e71c4e05ab4d7b90689c7991" member_type: VOTER last_known_addr { host: "127.3.240.253" port: 34933 } } peers { permanent_uuid: "c5d2b79be6a14f6eaad8cee88426a300" member_type: VOTER last_known_addr { host: "127.3.240.252" port: 39759 } } }
I20260810 11:43:42.832938 5393 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300 [sys.catalog]: This master's current role is: FOLLOWER
I20260810 11:43:42.855041 5449 catalog_manager.cc:1370] Generated new certificate authority record
I20260810 11:43:42.859755 5449 catalog_manager.cc:1504] Loading token signing keys...
I20260810 11:43:42.886494 5449 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f86c4679e0b42a0a3caad47d77a14a6: Generated new TSK 0
I20260810 11:43:42.887698 5449 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260810 11:43:43.044526 5388 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.026166 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49716 (local address 127.3.240.254:39571)
0810 11:43:43.026412 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.026418 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.042512 (+ 16094us) server_negotiation.cc:407] Connection header received
0810 11:43:43.042719 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.042726 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.042852 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.042990 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.044257 (+ 1267us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49716: BlockingRecv error: recv got EOF from 127.0.0.1:49716 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
W20260810 11:43:43.044957 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.032264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39888 (local address 127.3.240.253:34933)
0810 11:43:43.032630 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.032639 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.043177 (+ 10538us) server_negotiation.cc:407] Connection header received
0810 11:43:43.043449 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.043456 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.043548 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.043626 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.044726 (+ 1100us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39888: BlockingRecv error: recv got EOF from 127.0.0.1:39888 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
W20260810 11:43:43.047049 5387 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.042685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33406 (local address 127.3.240.252:39759)
0810 11:43:43.042918 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.042946 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.045510 (+ 2564us) server_negotiation.cc:407] Connection header received
0810 11:43:43.045737 (+ 227us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.045743 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.045825 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.045898 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.046814 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33406: BlockingRecv error: recv got EOF from 127.0.0.1:33406 (error 108)
Metrics: {"server-negotiator.queue_time_us":82}
W20260810 11:43:43.053890 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.051565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.3.240.253:34933)
0810 11:43:43.051787 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.051794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.052316 (+ 522us) server_negotiation.cc:407] Connection header received
0810 11:43:43.052554 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.052560 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.052674 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.052800 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.053621 (+ 821us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39894: BlockingRecv error: recv got EOF from 127.0.0.1:39894 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260810 11:43:43.061436 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.058664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39908 (local address 127.3.240.253:34933)
0810 11:43:43.058837 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.058843 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.059539 (+ 696us) server_negotiation.cc:407] Connection header received
0810 11:43:43.059764 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.059770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.059863 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.059979 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.061191 (+ 1212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39908: BlockingRecv error: recv got EOF from 127.0.0.1:39908 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260810 11:43:43.069612 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.066810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39920 (local address 127.3.240.253:34933)
0810 11:43:43.067046 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.067052 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.067598 (+ 546us) server_negotiation.cc:407] Connection header received
0810 11:43:43.067943 (+ 345us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.067949 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.068037 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.068130 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.069218 (+ 1088us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39920: BlockingRecv error: recv got EOF from 127.0.0.1:39920 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260810 11:43:43.080542 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.077839 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39934 (local address 127.3.240.253:34933)
0810 11:43:43.078090 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.078101 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.078684 (+ 583us) server_negotiation.cc:407] Connection header received
0810 11:43:43.078898 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.078903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.079023 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.079134 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.080258 (+ 1124us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39934: BlockingRecv error: recv got EOF from 127.0.0.1:39934 (error 108)
Metrics: {"server-negotiator.queue_time_us":104}
W20260810 11:43:43.095221 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.092699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39946 (local address 127.3.240.253:34933)
0810 11:43:43.092904 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.092911 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.093520 (+ 609us) server_negotiation.cc:407] Connection header received
0810 11:43:43.093723 (+ 203us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.093731 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.093861 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.093998 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.094914 (+ 916us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39946: BlockingRecv error: recv got EOF from 127.0.0.1:39946 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20260810 11:43:43.147269 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.144904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39954 (local address 127.3.240.253:34933)
0810 11:43:43.145104 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.145110 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.145719 (+ 609us) server_negotiation.cc:407] Connection header received
0810 11:43:43.145861 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.145866 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.145956 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.146062 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.147040 (+ 978us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39954: BlockingRecv error: recv got EOF from 127.0.0.1:39954 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 11:43:43.186462 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.183899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39966 (local address 127.3.240.253:34933)
0810 11:43:43.184117 (+ 218us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.184123 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.184472 (+ 349us) server_negotiation.cc:407] Connection header received
0810 11:43:43.184760 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.184765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.184923 (+ 158us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.185064 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.186144 (+ 1080us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39966: BlockingRecv error: recv got EOF from 127.0.0.1:39966 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260810 11:43:43.257879 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.255048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39978 (local address 127.3.240.253:34933)
0810 11:43:43.255292 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.255301 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.255773 (+ 472us) server_negotiation.cc:407] Connection header received
0810 11:43:43.255949 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.255955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.256054 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.256203 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.257619 (+ 1416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39978: BlockingRecv error: recv got EOF from 127.0.0.1:39978 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
W20260810 11:43:43.392976 5391 negotiation.cc:337] Failed RPC negotiation. Trace:
0810 11:43:43.390132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49644 (local address 127.3.240.253:34933)
0810 11:43:43.390379 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.390387 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.390761 (+ 374us) server_negotiation.cc:407] Connection header received
0810 11:43:43.391290 (+ 529us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.391295 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.391374 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.391483 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0810 11:43:43.392733 (+ 1250us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:49644: BlockingRecv error: recv got EOF from 127.0.0.1:49644 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
I20260810 11:43:43.417841 5410 catalog_manager.cc:1259] Loaded cluster ID: 5fc4ed2fd3ca4427a9b5f4d4bfaec402
I20260810 11:43:43.417945 5410 catalog_manager.cc:1552] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: loading cluster ID for follower catalog manager: success
I20260810 11:43:43.421218 5410 catalog_manager.cc:1574] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: acquiring CA information for follower catalog manager: success
I20260810 11:43:43.423492 5410 catalog_manager.cc:1602] T 00000000000000000000000000000000 P c5d2b79be6a14f6eaad8cee88426a300: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:43:43.504263 5429 catalog_manager.cc:1259] Loaded cluster ID: 5fc4ed2fd3ca4427a9b5f4d4bfaec402
I20260810 11:43:43.504609 5429 catalog_manager.cc:1552] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: loading cluster ID for follower catalog manager: success
I20260810 11:43:43.507520 5429 catalog_manager.cc:1574] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: acquiring CA information for follower catalog manager: success
I20260810 11:43:43.509466 5429 catalog_manager.cc:1602] T 00000000000000000000000000000000 P efc440c0e71c4e05ab4d7b90689c7991: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260810 11:43:43.661731 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:43.654002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49652 (local address 127.3.240.253:34933)
0810 11:43:43.654219 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.654224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.654572 (+ 348us) server_negotiation.cc:407] Connection header received
0810 11:43:43.654709 (+ 137us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.654714 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.654995 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.655257 (+ 262us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:43.655835 (+ 578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.656921 (+ 1086us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.659095 (+ 2174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.659555 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.659681 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:43.660400 (+ 719us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:43.661012 (+ 612us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:43.661213 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:43.661401 (+ 188us) server_negotiation.cc:299] Negotiation successful
0810 11:43:43.661509 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260810 11:43:43.677101 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:43.669060 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43404 (local address 127.3.240.252:39759)
0810 11:43:43.669544 (+ 484us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.669549 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.669765 (+ 216us) server_negotiation.cc:407] Connection header received
0810 11:43:43.670267 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.670272 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.670441 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.670604 (+ 163us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:43.671247 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.672225 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.674036 (+ 1811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.674583 (+ 547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.674717 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:43.675654 (+ 937us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:43.676172 (+ 518us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:43.676317 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:43.676555 (+ 238us) server_negotiation.cc:299] Negotiation successful
0810 11:43:43.676739 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":278,"threads_started":1}
I20260810 11:43:43.695098 5391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:43.684200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49666 (local address 127.3.240.253:34933)
0810 11:43:43.684400 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.684406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.684872 (+ 466us) server_negotiation.cc:407] Connection header received
0810 11:43:43.685238 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.685249 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.685391 (+ 142us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.685483 (+ 92us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:43.686812 (+ 1329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.688425 (+ 1613us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.692365 (+ 3940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.693179 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.693340 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:43.694245 (+ 905us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:43.694366 (+ 121us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:43.694476 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:43.694658 (+ 182us) server_negotiation.cc:299] Negotiation successful
0810 11:43:43.694754 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260810 11:43:43.699787 5570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:43.684889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43406 (local address 127.3.240.252:39759)
0810 11:43:43.685037 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.685042 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.693053 (+ 8011us) server_negotiation.cc:407] Connection header received
0810 11:43:43.693285 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.693291 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.693363 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.693476 (+ 113us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:43.694143 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.695389 (+ 1246us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.697656 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.698148 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.698273 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:43.698694 (+ 421us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:43.698827 (+ 133us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:43.698898 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:43.699270 (+ 372us) server_negotiation.cc:299] Negotiation successful
0810 11:43:43.699356 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260810 11:43:43.700937 5578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0810 11:43:43.684054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59754 (local address 127.3.240.254:39571)
0810 11:43:43.693160 (+ 9106us) server_negotiation.cc:206] Beginning negotiation
0810 11:43:43.693166 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0810 11:43:43.693191 (+ 25us) server_negotiation.cc:407] Connection header received
0810 11:43:43.693342 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0810 11:43:43.693348 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0810 11:43:43.693768 (+ 420us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0810 11:43:43.694016 (+ 248us) server_negotiation.cc:226] Negotiated authn=TOKEN
0810 11:43:43.694845 (+ 829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.695906 (+ 1061us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.698031 (+ 2125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0810 11:43:43.698561 (+ 530us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0810 11:43:43.698708 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0810 11:43:43.699545 (+ 837us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0810 11:43:43.700060 (+ 515us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0810 11:43:43.700177 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0810 11:43:43.700552 (+ 375us) server_negotiation.cc:299] Negotiation successful
0810 11:43:43.700649 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8926,"thread_start_us":8754,"threads_started":1}
I20260810 11:43:43.706897 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5065
I20260810 11:43:43.708227 5152 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:43.830572 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5065
W20260810 11:43:43.862819 5311 connection.cc:535] server connection from 127.0.0.1:33380 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260810 11:43:43.863317 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5168
I20260810 11:43:43.864544 5260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:43.982693 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5168
I20260810 11:43:44.012426 4035 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskagqWOB/build/asan/bin/kudu pid 5278
I20260810 11:43:44.013974 5377 generic_service.cc:196] Checking for leaks (request via RPC)
I20260810 11:43:44.126848 4035 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskagqWOB/build/asan/bin/kudu with pid 5278
2026-08-10T11:43:44Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5338 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (10686 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (414169 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST